A point cloud data processing method and related apparatus

Abstract

The application discloses a point cloud data processing method applied to the field of artificial intelligence technology. In the method, based on point cloud data and images collected for the same object, the object included in the images is recognized first, and then the point cloud cluster corresponding to the object in the images in the point cloud data is determined through the mapping relationship between the point cloud data and the images. In this way, in the model training stage, the point cloud cluster and the text features corresponding to the same object can be extracted through the model, comparison learning training of the point cloud data features and the text features is realized, and the subsequent recognition of the object in the point cloud data can be ensured based on the model obtained through the training.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method and related apparatus for processing point cloud data. Background Technology

[0002] The goal of 3D object localization and discrimination tasks is to locate and classify objects in a 3D scene constructed from point cloud data. With the continuous development of intelligent robots and autonomous driving, the localization and recognition of 3D objects in open-world environments has gradually become an important perception objective.

[0003] With the rapid development of artificial intelligence (AI) technology, AI-based models have achieved good results in locating and recognizing 3D objects. However, locating and recognizing 3D objects often requires training the model with a large amount of labeled data to achieve good performance. Due to the vastness of open-world 3D scenes, practical applications often only bear the cost of labeling a portion of the data, limiting the training of 3D object localization and recognition tasks in open-world semantics. Therefore, how to build localization and recognition models for open worlds in the absence of labeled data has become a key development trend in the industry.

[0004] Currently, related technologies project point cloud data into multiple depth maps, and then input these depth maps and text vocabulary into the image feature extractor and text feature extractor of the model, respectively, for feature extraction. This allows for comparative learning and training of image and text features. During model testing, the depth maps projected from the point cloud data and the test text are input into the model, and the feature distances between the model's output image features and each text feature are calculated to determine the text corresponding to the point cloud data, thus achieving object recognition in the point cloud data. However, these technologies require projecting point cloud data into depth maps, which often results in the loss of a significant amount of the original 3D data structure, potentially leading to poor actual model performance. Summary of the Invention

[0005] This application provides a method for processing point cloud data, which can effectively improve the performance of models used to achieve object recognition in point cloud data.

[0006] This application provides a method for processing point cloud data, applicable to the field of artificial intelligence technology. The method includes: firstly, acquiring first point cloud data and a first image, wherein the first point cloud data and the first image are obtained by collecting data from a first object. Furthermore, in addition to the first object, the first point cloud data and the first image may also include other objects, i.e., the first point cloud data and the first image may include multiple objects.

[0007] Then, the image block of the first object in the first image and the text name corresponding to the first object are obtained, that is, the alignment of the image and the text is realized, and the text semantics of the image block in the first image are determined.

[0008] Secondly, based on the mapping relationship between the first point cloud data and the first image, and the image blocks, the first point cloud cluster corresponding to the first object in the first point cloud data is determined. This mapping relationship enables the conversion between points in the first point cloud data and pixels in the first image. In other words, based on this mapping relationship, it is possible to determine the pixels corresponding to points in the first point cloud data in the first image, and vice versa.

[0009] Next, features of the first point cloud cluster and the text name are extracted using the first model to obtain the first point cloud features and the first text features. The first model includes a first network and a second network. The first point cloud features are obtained by the first network extracting features from the first point cloud cluster, and the first text features are obtained by the second network extracting features from the text name. The first network and the second network are structurally different and independent neural networks.

[0010] Finally, the first model is updated based on the first loss function to obtain the second model, where the first loss function is related to the first difference between the first point cloud features and the second text features. This first loss function can be positively correlated with the first difference between the first point cloud features and the second text features. Thus, in the process of updating the first model based on the first loss function, the update objective of the first model is to minimize the first loss function, that is, to reduce the difference between the first point cloud features and the first text features corresponding to the same object. This ensures that the trained second model, when extracting point cloud features and text features corresponding to the same object, can output point cloud features and text features with the smallest possible differences.

[0011] In this scheme, based on point cloud data and images collected from the same object, the objects included in the image are first identified. Then, through the mapping relationship between the point cloud data and the image, the point cloud clusters corresponding to the objects in the image are determined. In this way, during the model training phase, the features of the point cloud clusters corresponding to the same object and the text can be extracted by the model, realizing comparative learning training of point cloud data features and text features, ensuring that the trained model can subsequently identify objects in the point cloud data.

[0012] Since this solution uses images as an intermediary between point cloud data and text, the text corresponding to the point cloud data is determined in advance through the mapping relationship between point cloud data and images during the acquisition stage, thereby achieving alignment between point cloud data and text. This avoids the loss of the original data structure caused by projecting point cloud data into a depth map, which is beneficial for understanding objects in 3D scenes and can effectively improve the performance of the model.

[0013] In one possible implementation, the method further includes: acquiring second point cloud features and second text features. The second point cloud features are obtained by the first network extracting features from the point cloud clusters corresponding to the second object, and the second text features are obtained by the second network extracting features from the text name of the second object. The first object and the second object are different objects. For example, the first object and the second object can be different objects belonging to the same first image; the second object can also be an object in other images, such as objects in other images in the same batch of training data as the first image.

[0014] The first loss function is related to the first difference and the second difference, where the second difference is the difference between the first text feature and the second point cloud feature.

[0015] In one possible implementation, the first loss function is positively correlated with the first difference, and the first loss function is negatively correlated with the second difference.

[0016] In this scheme, a first loss function is constructed based on both the first and second differences. This allows the first model to learn, during training, to simultaneously learn to bring point cloud features and text features corresponding to the same object closer together, and to learn to widen the gaps between point cloud features and text features corresponding to different objects. Consequently, after training, the second model can output point cloud features and text features with the smallest possible differences when extracting point cloud features and text features corresponding to the same object; and it can output point cloud features and text features with the largest possible differences when extracting point cloud features and text features corresponding to different objects. This results in better performance when using the second model to identify the category of point cloud data.

[0017] In one possible implementation, the first model may further include a third network. This third network is used to extract features from the image; that is, the input to the third network is the image, and the output is the image's features. The third network is a structurally different and independent neural network from the first and second networks described above. This third network could, for example, be an attention network.

[0018] The method further includes: extracting features from image patches through a third network to obtain first image features. The first loss function is related to a first difference and a third difference, where the third difference is the difference between the first image features and the first point cloud features.

[0019] In this scheme, the loss function is constructed by introducing the differences between point cloud features and text features, as well as between point cloud features and image features. This enables the model to learn to approximate point cloud features, text features, and image features corresponding to the same object as much as possible during the training process based on the loss function, which can effectively improve the performance of the finally trained model.

[0020] In one possible implementation, the method further includes: acquiring second point cloud data, which is point cloud data for object recognition to be performed.

[0021] The second point cloud data is input into the first network in the second model to obtain the third point cloud features;

[0022] Multiple texts are input into the second network of the second model to obtain multiple text features. Each text feature corresponds one-to-one with a specific text. Furthermore, these multiple texts can be determined based on the acquisition scenario of the second point cloud data. For example, in an autonomous driving scenario, the multiple texts could include words such as "car," "truck," "bicycle," "pedestrian," "roadblock," and "tree."

[0023] Based on the difference between the third point cloud feature and each of the multiple text features, the target text corresponding to the second point cloud data is determined, where the target text corresponds to the target text feature, which is the text feature with the smallest difference from the third point cloud feature among the multiple text features.

[0024] In other words, after determining the difference between the third point cloud feature and each of the multiple text features, the differences between the third point cloud feature and each text feature can be sorted in ascending order, and the text feature with the smallest difference from the third point cloud feature (i.e., the target text feature mentioned above) can be determined. Then, the text corresponding to this text feature is taken as the target text.

[0025] In one possible implementation, the above mapping relationship is obtained based on the device for acquiring the first point cloud data and the device for acquiring the first image. That is, the device for acquiring the first point cloud data and the device for acquiring the first image affect the conversion relationship between the first point cloud data and the first image.

[0026] In one possible implementation, the first point cloud data is acquired by LiDAR, the first image is acquired by an image sensor, and the LiDAR and the image sensor are deployed on the same device.

[0027] In one possible implementation, based on the mapping relationship between the first point cloud data and the first image, and the image patch, the first point cloud cluster corresponding to the first object in the first point cloud data is determined, including: based on the coordinate system transformation matrix between the lidar and the image sensor, the second point cloud cluster corresponding to the image patch in the first point cloud data is determined; the points in the second point cloud cluster are clustered to obtain the first point cloud cluster, wherein the second point cloud cluster includes the first point cloud cluster.

[0028] In this scheme, by performing clustering processing on the points in the second point cloud cluster, the points belonging to the foreground object in the second point cloud cluster can be effectively extracted, that is, the points belonging to the first object can be extracted, thereby removing the points that do not belong to the first object and ensuring the accuracy of the final first point cloud cluster.

[0029] In one possible implementation, the first point cloud data and the first image are acquired using the same depth camera.

[0030] In one possible implementation, the first point cloud cluster corresponding to the first object in the first point cloud data is determined based on the mapping relationship between the first point cloud data and the first image and the image patch, including: extracting the foreground part in the image patch to obtain the foreground image patch; and determining the first point cloud cluster corresponding to the foreground image patch in the first point cloud data based on the coordinate system transformation matrix of the depth camera.

[0031] In this scheme, by first extracting the foreground part of the image and then converting the foreground part of the image into a point cloud cluster, other objects besides the first object can be effectively extracted before the image is converted into a point cloud cluster, ensuring the accuracy of the final first point cloud cluster.

[0032] In one possible implementation, obtaining the image patch of the first object in the first image and the text name corresponding to the first object includes: performing target recognition on the first image through a second model to obtain the image patch of the first object in the first image and the text name corresponding to the first object, wherein the second model is a pre-trained model.

[0033] A second aspect of this application provides a point cloud data processing apparatus, comprising:

[0034] The acquisition module is used to acquire the first point cloud data and the first image, which are obtained by collecting data from the first object.

[0035] The acquisition module is also used to acquire the image block of the first object in the first image and the text name corresponding to the first object;

[0036] The processing module is used to determine the first point cloud cluster corresponding to the first object in the first point cloud data based on the mapping relationship between the first point cloud data and the first image and the image blocks;

[0037] The processing module is also used to extract features of the first point cloud cluster and the text name through the first model to obtain the first point cloud features and the first text features. The first model includes a first network and a second network. The first point cloud features are obtained by the first network extracting features from the first point cloud cluster, and the first text features are obtained by the second network extracting features from the text name.

[0038] The processing module is also used to update the first model based on the first loss function to obtain the second model, wherein the first loss function is related to the first difference between the first point cloud features and the second text features.

[0039] In one possible implementation, the acquisition module is also used to acquire second point cloud features and second text features. The second point cloud features are obtained by the first network extracting features from the point cloud clusters corresponding to the second object, and the second text features are obtained by the second network extracting features from the text name of the second object. The first object and the second object are different objects.

[0040] The first loss function is related to the first difference and the second difference, where the second difference is the difference between the first text feature and the second point cloud feature.

[0041] In one possible implementation, the first loss function is positively correlated with the first difference, and the first loss function is negatively correlated with the second difference.

[0042] In one possible implementation, the first model also includes a third network;

[0043] The processing module is also used for:

[0044] The first image features are obtained by extracting features from image patches using a third network;

[0045] The first loss function is related to the first difference and the third difference, where the third difference is the difference between the first image features and the first point cloud features.

[0046] In one possible implementation, the acquisition module is also used to acquire second point cloud data;

[0047] The processing module is also used to input the second point cloud data into the first network in the second model to obtain the third point cloud features;

[0048] The processing module is also used to input multiple texts into the second network of the second model to obtain multiple text features;

[0049] The processing module is also used to determine the target text corresponding to the second point cloud data based on the difference between the third point cloud feature and each of the multiple text features, wherein the target text corresponds to the target text feature, and the target text feature is the text feature with the smallest difference from the third point cloud feature among the multiple text features.

[0050] In one possible implementation, the mapping relationship is obtained based on the device for acquiring the first point cloud data and the device for acquiring the first image.

[0051] In one possible implementation, the first point cloud data is acquired by LiDAR, the first image is acquired by an image sensor, and the LiDAR and the image sensor are deployed on the same device.

[0052] In one possible implementation, the processing module is also used for:

[0053] Based on the coordinate system transformation matrix between the lidar and the image sensor, the second point cloud cluster corresponding to the image patch in the first point cloud data is determined.

[0054] Clustering is performed on the points in the second point cloud cluster to obtain the first point cloud cluster, where the second point cloud cluster includes the first point cloud cluster.

[0055] In one possible implementation, the first point cloud data and the first image are acquired using the same depth camera.

[0056] In one possible implementation, the processing module is also used for:

[0057] Extract the foreground portion from the image patch to obtain the foreground image patch;

[0058] Based on the coordinate system transformation matrix of the depth camera, the first point cloud cluster corresponding to the foreground image patch in the first point cloud data is determined.

[0059] In one possible implementation, the processing module is also used for:

[0060] The second model is used to perform target recognition on the first image to obtain the image patch of the first object in the first image and the text name corresponding to the first object. The second model is a pre-trained model.

[0061] A third aspect of this application provides a point cloud data processing apparatus, which may include a processor and a memory coupled together. The memory stores program instructions, and when the program instructions stored in the memory are executed by the processor, the method described in the first aspect or any implementation thereof is implemented. For details regarding the steps in the various possible implementations of the first aspect executed by the processor, please refer to the first aspect; further details will not be repeated here.

[0062] The fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to perform the method of any implementation of the first aspect described above.

[0063] The fifth aspect of this application provides a circuit system including a processing circuit configured to perform the method of any implementation of the first aspect described above.

[0064] The sixth aspect of this application provides a computer program product that, when run on a computer, causes the computer to perform any implementation of the first aspect described above.

[0065] A seventh aspect of this application provides a chip system including a processor for supporting a server or threshold value acquisition device in implementing the functions involved in any implementation of the first aspect described above, such as transmitting or processing data and / or information involved in the methods described above. In one possible design, the chip system further includes a memory for storing program instructions and data necessary for the server or communication device. This chip system may be composed of chips or may include chips and other discrete devices.

[0066] The beneficial effects of the second to seventh aspects mentioned above can be referred to the introduction of the first aspect above, and will not be repeated here. Attached Figure Description

[0067] Figure 1 This is a schematic diagram of the structure of a convolutional neural network provided in an embodiment of this application;

[0068] Figure 2 This is a schematic diagram of another convolutional neural network structure provided in an embodiment of this application;

[0069] Figure 3 This is a schematic diagram of a process for processing point cloud data in a related technology.

[0070] Figure 4 A schematic diagram of a system architecture 400 provided for an embodiment of this application;

[0071] Figure 5 This is a schematic diagram of the structure of an electronic device 101 provided in an embodiment of this application;

[0072] Figure 6 A flowchart illustrating a point cloud data processing method provided in an embodiment of this application;

[0073] Figure 7 This application provides a schematic diagram of an embodiment of recognizing objects in an image using a second model.

[0074] Figure 8A schematic diagram of a system architecture provided for an embodiment of this application;

[0075] Figure 9A This application provides an embodiment of object recognition in an outdoor open world.

[0076] Figure 9B This application provides an embodiment of object recognition in an indoor open world.

[0077] Figure 10 This is a schematic diagram illustrating a process for locating and recognizing 3D objects in an open world, provided as an embodiment of this application.

[0078] Figure 11 A schematic diagram illustrating the process of trimodal data alignment in an outdoor scenario, provided as an embodiment of this application;

[0079] Figure 12 A schematic diagram illustrating the process of trimodal data alignment in an indoor scene, provided as an embodiment of this application;

[0080] Figure 13 This is a schematic diagram illustrating a process for performing contrastive pre-training on text, image, and point cloud data using a three-modal contrastive pre-training three-tower model, as provided in an embodiment of this application.

[0081] Figure 14 This is a schematic diagram of a three-modal joint feature discrimination process provided in an embodiment of this application;

[0082] Figure 15 This is a schematic diagram of the structure of a point cloud data processing device provided in an embodiment of this application;

[0083] Figure 16 A schematic diagram of the structure of the execution device provided in the embodiments of this application;

[0084] Figure 17 A schematic diagram of the structure of a chip provided in an embodiment of this application;

[0085] Figure 18 This is a schematic diagram of the structure of a computer-readable storage medium provided in an embodiment of this application. Detailed Implementation

[0086] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0087] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0088] For ease of understanding, some technical terms involved in the embodiments of this application will be introduced below.

[0089] (1) Open-World

[0090] An open world refers to a generalized open scene that does not provide prior environmental annotations, that is, an open scene where the objects appearing in the environment are not labeled.

[0091] (2) Point cloud data

[0092] Point cloud data refers to a collection of vectors in a three-dimensional coordinate system. Typically, point cloud data is obtained by scanning the environment using a laser scanner (such as LiDAR). Furthermore, point cloud data is recorded in the form of points, each containing three-dimensional coordinates. In some cases, the points in the point cloud data may contain color information or reflection intensity information. The reflection intensity information refers to the echo intensity collected by the laser scanner's receiving device; this reflection intensity information is related to the target's surface material, roughness, incident angle, and the instrument's emission energy and laser wavelength.

[0093] (3) Laser Radar

[0094] LiDAR (Light Detection and Ranging) is a radar system that uses emitted laser beams to detect the position, velocity, and other characteristics of targets. The working principle of LiDAR is to emit a detection signal (laser beam) towards the target, then compare and process the received signal reflected back from the target (target echo) with the emitted signal to obtain relevant information about the target, such as its distance, azimuth, altitude, velocity, attitude, and even shape. This allows for the detection of targets on the ground, such as roads or obstacles. Generally, LiDAR consists of a laser transmitter, an optical receiver, and an information processing system. The laser transmitter converts electrical pulses into light pulses and emits them. The optical receiver then converts the light pulses reflected from the target back into electrical pulses, which are then sent to the information processing system for processing.

[0095] (4) Image sensor

[0096] An image sensor utilizes the photoelectric conversion function of optoelectronic devices to convert the light image on a photosensitive surface into an electrical signal proportional to the light image. Compared to photosensitive elements with "point" light sources such as photodiodes and phototransistors, an image sensor is a functional device that divides the light image on its light-receiving surface into many small units and converts them into usable electrical signals. Simply put, an image sensor is a device used to acquire images.

[0097] (5) Depth camera

[0098] A depth camera is a type of camera that can simultaneously acquire an image of a target and the distance between the target and the depth camera. Based on a depth camera, the distance between each pixel in the image and the depth camera can be obtained. By adding the distance of that pixel to the two-dimensional coordinates of the image, the three-dimensional spatial coordinates of each pixel in the image can be obtained.

[0099] In other words, compared to traditional cameras, depth cameras can not only acquire images, but also the spatial distances between various targets in the image and the depth camera.

[0100] (6) Neural Networks

[0101] A neural network can be composed of neural units, which can be defined as a computational unit that takes xs (i.e., input data) and an intercept of 1 as input. The output of this computational unit can be:

[0102]

[0103] Where s = 1, 2, ..., n, where n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into an output signal. The output signal of this activation function can be used as the input of the next convolutional layer, and the activation function can be the sigmoid function. A neural network is a network formed by connecting multiple of the above-mentioned individual neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, which can be a region composed of several neural units.

[0104] (7) Deep Neural Network (DNN)

[0105] Deep neural networks, also known as multilayer neural networks, can be understood as neural networks with many hidden layers, though there's no specific metric for "many." DNNs can be categorized into three types based on their layer positions: input layers, hidden layers, and output layers. Generally, the first layer is the input layer, the last layer is the output layer, and the layers in between are hidden layers. All layers are fully connected, meaning that any neuron in the i-th layer is connected to any neuron in the (i+1)-th layer. Although DNNs appear complex, the operation of each layer is actually quite simple, resembling a linear relationship as follows: in, It is the input vector. It is the output vector. α is the offset vector, W is the weight matrix (also called coefficients), and α() is the activation function. Each layer is simply an adjustment of the input vector. The output vector is obtained through such a simple operation. Because DNNs have many layers, the coefficients W and the offset vector... The number of these parameters is therefore quite large. The definitions of these parameters in a DNN are as follows: Taking the coefficient W as an example: Assuming a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as... The superscript 3 represents the layer number where coefficient W resides, while the subscript corresponds to the output third layer index 2 and the input second layer index 4. In summary, the coefficients from the k-th neuron in layer L-1 to the j-th neuron in layer L are defined as follows: It's important to note that the input layer does not have a W parameter. In deep neural networks, more hidden layers allow the network to better represent complex real-world situations. Theoretically, the more parameters a model has, the higher its complexity and "capacity," meaning it can perform more complex learning tasks. Training a deep neural network is essentially the process of learning the weight matrix, with the ultimate goal of obtaining the weight matrix of all layers in the trained deep neural network (a weight matrix formed by the vectors W from many layers).

[0106] (8) Convolutional Neural Network (CNN)

[0107] A convolutional neural network (CNN) is a deep neural network with a convolutional structure. A CNN contains a feature extractor consisting of convolutional layers and subsampling layers. This feature extractor can be viewed as a filter, and the convolution process can be seen as using a trainable filter to convolve with an input image or feature map. A convolutional layer refers to the layer of neural units in a CNN that performs convolution processing on the input signal (e.g., the first and second convolutional layers in this embodiment). In a convolutional layer of a CNN, a neural unit can be connected to only some neighboring neural units. A convolutional layer typically contains several feature planes, each of which can be composed of a series of rectangularly arranged neural units. Neural units on the same feature plane share weights, which are the convolutional kernel. Shared weights can be understood as the way image information is extracted regardless of location. The underlying principle is that the statistical information of one part of the image is the same as that of other parts. This means that image information learned in one part can also be used in another part. Therefore, for all locations on the image, we can use the same learned image information. In the same convolutional layer, multiple convolutional kernels can be used to extract different image information. Generally, the more convolutional kernels there are, the richer the image information reflected by the convolution operation.

[0108] Convolutional kernels can be initialized as matrices of random size, and during the training of a convolutional neural network, they can learn appropriate weights. Furthermore, sharing weights directly reduces the number of connections between layers in the convolutional neural network, while also lowering the risk of overfitting.

[0109] Specifically, such as Figure 1 As shown, the convolutional neural network (CNN) 100 may include an input layer 110, a convolutional / pooling layer 120, wherein the pooling layer is optional, and a neural network layer 130.

[0110] The structure consisting of the convolutional layer / pooling layer 120 and the neural network layer 130 can be the first convolutional layer and the second convolutional layer described in this application. The input layer 110 is connected to the convolutional layer / pooling layer 120, and the convolutional layer / pooling layer 120 is connected to the neural network layer 130. The output of the neural network layer 130 can be input to the activation layer, and the activation layer can perform non-linear processing on the output of the neural network layer 130.

[0111] Convolutional / pooling layers: 120. Convolutional layers: such as... Figure 2The convolutional / pooling layer 120 shown may include layers 121-126 as in Examples 121-126. In one implementation, layer 121 is a convolutional layer, layer 122 is a pooling layer, layer 123 is a convolutional layer, layer 124 is a pooling layer, layer 125 is a convolutional layer, and layer 126 is a pooling layer. In another implementation, layers 121 and 122 are convolutional layers, layer 123 is a pooling layer, layers 124 and 125 are convolutional layers, and layer 126 is a pooling layer. That is, the output of the convolutional layer can be used as the input of a subsequent pooling layer, or as the input of another convolutional layer to continue the convolution operation.

[0112] Taking convolutional layer 121 as an example, it can include multiple convolution operators, also known as kernels. In image processing, a convolution operator acts as a filter, extracting specific information from the input image matrix. Essentially, a convolution operator can be a weight matrix, which is usually predefined. During the convolution operation, the weight matrix processes the input image pixel by pixel (or two pixels by two pixels, depending on the stride) along the horizontal direction, thus extracting specific features. The size of the weight matrix should be related to the image size. It's important to note that the depth dimension of the weight matrix is ​​the same as the depth dimension of the input image; during convolution, the weight matrix extends to the entire depth of the input image. Therefore, convolution with a single weight matrix produces a single-depth convolutional output. However, in most cases, multiple weight matrices of the same dimension are applied instead of a single weight matrix. The outputs of each weight matrix are stacked to form the depth dimension of the convolutional image. Different weight matrices can be used to extract different features from an image. For example, one weight matrix can be used to extract image edge information, another weight matrix can be used to extract specific colors from the image, and yet another weight matrix can be used to blur unwanted noise in the image. These multiple weight matrices have the same dimension, and the feature maps extracted by these multiple weight matrices with the same dimension also have the same dimension. The extracted feature maps with the same dimension are then merged to form the output of the convolution operation.

[0113] The weight values ​​in these weight matrices need to be obtained through extensive training in practical applications. The weight matrices formed by the weight values ​​obtained through training can extract information from the input image, thereby helping the convolutional neural network 100 to make correct predictions.

[0114] When a convolutional neural network 100 has multiple convolutional layers, the initial convolutional layers (e.g., 121) tend to extract more general features, which can also be called low-level features. As the depth of the convolutional neural network 100 increases, the features extracted by later convolutional layers (e.g., 126) become more and more complex, such as high-level semantic features. Features with higher semantic levels are more suitable for the problem to be solved.

[0115] Pooling layers: Because it is often necessary to reduce the number of training parameters, pooling layers are often introduced periodically after convolutional layers, such as... Figure 1 In the example of 120, each layer 121-126 can be a convolutional layer followed by a pooling layer, or multiple convolutional layers followed by one or more pooling layers.

[0116] Neural Network Layer 130: After processing by the convolutional / pooling layers 120, the convolutional neural network 100 is still insufficient to output the required output information. As before, the convolutional / pooling layers 120 only extract features and reduce the parameters introduced by the input image. However, to generate the final output information (the required class information or other relevant information), the convolutional neural network 100 needs to utilize neural network layer 130 to generate one or a set of required class numbers of output. Therefore, neural network layer 130 may include multiple hidden layers (such as...). Figure 1 As shown in 131, 132 to 13n) and output layer 140, the parameters contained in these multi-layer hidden layers can be pre-trained based on relevant training data for specific task types, such as image recognition, image classification, image super-resolution reconstruction, etc.

[0117] After the multiple hidden layers in neural network layer 130, the final layer of the entire convolutional neural network 100 is the output layer 140. This output layer 140 has a loss function similar to classification cross-entropy, specifically used to calculate the prediction error. Once the entire convolutional neural network 100 has undergone forward propagation (e.g., ...), the loss function is applied. Figure 2 The propagation from 110 to 140 is completed (forward propagation), and the reverse propagation (such as...) Figure 2 The propagation from 140 to 110 (backpropagation) will begin to update the weight values ​​and biases of the layers mentioned above, in order to reduce the loss of the convolutional neural network 100 and the error between the output of the convolutional neural network 100 through the output layer and the ideal result.

[0118] It should be noted that, as Figure 1 The convolutional neural network 100 shown is merely an example of a convolutional neural network. In specific applications, convolutional neural networks can also exist in the form of other network prediction models, such as... Figure 2The multiple convolutional / pooling layers shown are run in parallel, and the extracted features are all input into the full neural network layer 130 for processing.

[0119] (9) Attention Network

[0120] Attention networks are network models that utilize attention mechanisms to improve model training speed. Currently, typical attention networks include the Transformer model. Models applying attention mechanisms can assign different weights to each part of the input sequence, thereby extracting more important feature information from the input sequence, resulting in a more accurate output.

[0121] In deep learning, attention mechanisms can be implemented using weight vectors that describe importance: when predicting or inferring an element, the weight vectors determine the correlation between that element and other elements. For example, for a pixel in an image or a word in a sentence, attention vectors can be used to quantitatively estimate the correlation between the target element and other elements, and the weighted sum of the attention vectors serves as an approximation of the target value.

[0122] The attention mechanism in deep learning simulates the human brain's attention mechanism. For example, when a human views a painting, although the eyes can see the entire picture, upon closer inspection, the eyes actually focus on only a portion of the image. At this point, the brain primarily focuses on this smaller area. In other words, when humans carefully observe an image, the brain's attention to the entire image is not uniform but rather differentiated by weight; this is the core idea of ​​the attention mechanism.

[0123] In simple terms, the human visual processing system often selectively focuses on certain parts of an image while ignoring other irrelevant information, thus aiding the brain's perception. Similarly, in the attention mechanism of deep learning, in problems involving language, speech, or vision, certain parts of the input may be more relevant than others. Therefore, through the attention mechanism in attention models, different processing can be applied to different parts of the input data, allowing the attention model to dynamically focus only on task-relevant data.

[0124] (10) Multilayer Perceptron (MLP)

[0125] A multilayer perceptron is a feedforward artificial neural network model that maps multiple input datasets to a single output dataset. A typical multilayer perceptron consists of three network layers: an input layer, hidden layers, and an output layer. Furthermore, in a multilayer perceptron, the different network layers are fully connected (i.e., any neuron in one layer is connected to all neurons in the next layer).

[0126] (11) Loss Function

[0127] During neural network training, to ensure the output closely approximates the desired predicted value, we compare the network's prediction with the target value. Based on the difference, we update the weight vector of each layer (usually pre-configuring parameters before the initial update). For example, if the prediction is too high, the weight vector is adjusted to predict a lower value. This adjustment continues until the neural network predicts the target value or a value very close to it. Therefore, we need to predefine "how to compare the difference between the predicted and target values," which is the loss function or objective function. These are important equations used to measure the difference between the predicted and target values. Taking the loss function as an example, a higher output value (loss) indicates a greater difference, and training the neural network becomes a process of minimizing this loss.

[0128] (12) Backpropagation algorithm

[0129] Neural networks can employ backpropagation (BP) to correct the parameters of the initial prediction model during training, thereby reducing the error loss. Specifically, forward propagation of the input signal to the output generates error loss; this error loss information is then propagated back to update the parameters of the initial prediction model, leading to convergence of the error loss. The backpropagation algorithm is an error-loss-driven backpropagation process aimed at obtaining the optimal parameters of the prediction model, such as the weight matrix.

[0130] (13) Gradient descent

[0131] Gradient descent is a first-order optimization algorithm commonly used in machine learning to recursively approximate a minimum-bias prediction model. To find a local minimum of a function using gradient descent, iterative searches must be performed at points a predetermined step distance away in the opposite direction of the gradient (or approximate gradient) at the current point on the function. Gradient descent is one of the most frequently used methods for solving the prediction model parameters of machine learning algorithms, i.e., unconstrained optimization problems.

[0132] Specifically, when finding the minimum value of the loss function, gradient descent can be used to iteratively solve the problem step by step, obtaining the minimized loss function and the predicted model parameter values. Conversely, if we need to find the maximum value of the loss function, we need to use gradient ascent iteratively.

[0133] (14) Softmax function

[0134] The Softmax function, also known as the normalized exponential function, is a generalization of the logistic function. The Softmax function transforms a K-dimensional vector Z containing arbitrary real numbers into another K-dimensional vector σ(Z), such that every element in the transformed vector σ(Z) is within the range (0, 1), and the sum of all elements is 1. The Softmax function can be calculated as shown in Equation 1.

[0135]

[0136] Where, σ(z) j Zj represents the value of the j-th element in the vector after the Softmax function transformation; Zk represents the value of the j-th element in vector Z; ∑ represents the summation.

[0137] (15) Pre-trained model

[0138] A pre-trained model is a pre-trained and saved network that has already been trained on a large dataset.

[0139] (16) Density-Based Spatial Clustering of Applications with Noise (DBSCAN)

[0140] DBSCAN is a representative density-based clustering algorithm. Unlike partitioning and hierarchical clustering methods, it defines a cluster as the largest set of density-connected points, enabling it to divide regions with sufficiently high density into clusters and discover clusters of arbitrary shapes in noisy spatial databases.

[0141] (17) Open domain vocabulary

[0142] Open-domain lexicon is a generalized dictionary based on a universal understanding of the world.

[0143] With the rapid development of artificial intelligence (AI) technology, using AI-based models (such as attention networks) to locate and recognize 3D objects has yielded good results. However, when using models to locate and recognize 3D objects, it is often necessary to first train the model with a large amount of labeled data (i.e., the point cloud data to be processed is labeled with the categories of each object) to achieve good model performance. Because the 3D scenes in open worlds are so vast, practical applications often can only afford the cost of labeling a portion of the data, limiting the training of 3D object localization and recognition tasks in open-world semantics. Therefore, how to build localization and recognition models for open worlds in the absence of labeled data has become a development trend of concern in the industry.

[0144] For example, please refer to Figure 3 , Figure 3 This is a schematic diagram of a process for processing point cloud data in related technologies. For example... Figure 3 As shown, the relevant technology first involves manually selecting point cloud data corresponding to a certain object and then projecting the point cloud data into multiple depth maps according to multiple camera positions. For example, the point cloud data corresponding to an aircraft is projected into depth maps at multiple angles.

[0145] Then, multiple depth maps and their corresponding text terms are input into the image feature extractor and text feature extractor in the model, respectively, for feature extraction. After multiple image features are extracted by the image feature extractor, these features are input into an adaptation network consisting of multiple fully convolutional layers. The adaptation network outputs fused features from multiple viewpoints to perform comparative learning training of the fused features and text features.

[0146] During model testing, the depth map projected from the point cloud data and the test text are input into the model. The feature distances between the output image features and various text features are calculated to determine the text corresponding to the point cloud data, thus achieving object recognition in the point cloud data. However, related technologies require projecting point cloud data into a depth map, which often results in the loss of a large amount of the original 3D data structure, easily leading to poor actual model performance. Furthermore, object recognition in point cloud data is based on the input of point cloud data containing only a single object (i.e., the input data is point cloud data containing only a single object), making it difficult to generalize to 3D scene understanding.

[0147] In view of this, embodiments of this application provide a method for processing point cloud data. Based on point cloud data and images collected from the same object, the method first identifies the objects included in the image, and then determines the point cloud clusters in the point cloud data corresponding to the objects in the image through the mapping relationship between the point cloud data and the image. Thus, during the model training phase, the model can extract features of the point cloud clusters corresponding to the same object and the text, achieving comparative learning training of point cloud data features and text features. This ensures that subsequent object recognition in point cloud data can be achieved based on the trained model. Since this solution uses images as an intermediary between point cloud data and text, the text corresponding to the point cloud data is determined in advance through the mapping relationship between the point cloud data and the image during the acquisition phase, thereby achieving alignment between point cloud data and text. This avoids the loss of the original data structure caused by projecting point cloud data into a depth map, which is beneficial for understanding objects in 3D scenes and can effectively improve model performance.

[0148] To facilitate understanding, the following will first introduce the application scenarios and system architecture of the point cloud data processing method provided in the embodiments of this application.

[0149] In one possible implementation, the point cloud data processing method provided in this embodiment can be applied to autonomous driving scenarios. For example, a point cloud data acquisition device and an image acquisition device are deployed on an autonomous vehicle. During operation, the autonomous vehicle acquires point cloud data and images of the same object and uploads the acquired data to a server. The server processes the point cloud data and images using the point cloud data processing method provided in this embodiment to train a model capable of recognizing objects in the point cloud data. Then, the server distributes the trained model to the autonomous vehicle, which uses this model to recognize objects in the open world during autonomous driving, enabling the vehicle to execute corresponding autonomous driving strategies based on the object recognition results.

[0150] Furthermore, after the autonomous vehicle collects point cloud data and images, it may not upload the point cloud data and images to the server. Instead, the autonomous vehicle itself processes the point cloud data and images using the point cloud data processing method provided in this embodiment to train a model capable of performing object recognition in the point cloud data. In other words, the autonomous vehicle no longer needs to interact with the server; instead, the autonomous vehicle completes the training and application of the model.

[0151] In another possible implementation, the point cloud data processing method provided in this embodiment can be applied to robot object recognition scenarios. For example, a point cloud data acquisition device and an image acquisition device are deployed on an indoor or outdoor robot. During its operation, the robot acquires point cloud data and images of the same object and uploads the acquired data and images to a server. The server processes the point cloud data and images using the point cloud data processing method provided in this embodiment to train a model capable of performing object recognition in the point cloud data. Then, the server distributes the trained model to the robot, which uses this model to recognize objects in the open world (e.g., indoor objects or outdoor obstacles) during its operation, enabling the robot to execute corresponding obstacle avoidance strategies based on the object recognition results.

[0152] Similarly, after the robot collects point cloud data and images, it may not upload the point cloud data and images to the server. Instead, the robot itself processes the point cloud data and images using the point cloud data processing method provided in this embodiment to train a model that can be used to perform object recognition in the point cloud data. In other words, the robot no longer needs to interact with the server; instead, the robot completes the training and application of the model.

[0153] In another possible implementation, the point cloud data processing method provided in this embodiment can be applied to object recognition scenarios of UAVs. For example, a point cloud data acquisition device and an image acquisition device are deployed on the UAV. During flight, the UAV acquires point cloud data and images of the same object and uploads the acquired point cloud data and images to a server. The server processes the point cloud data and images using the point cloud data processing method provided in this embodiment to train a model capable of performing object recognition in the point cloud data. Then, the server distributes the trained model to the UAV, which uses the model to identify objects in the open world (such as trees, houses, and other obstacles encountered during flight) during flight, thereby enabling the UAV to execute corresponding obstacle avoidance strategies based on the object recognition results.

[0154] Furthermore, when users control the drone via remote control, smartphones, tablets, laptops, or personal computers, the drone can also return point cloud data and images acquired through point cloud data acquisition and image acquisition devices to the user device. In this way, the user can train a model using the point cloud data and images acquired on the user device; alternatively, the user can send point cloud data and images to a server, which will then train a model and send it back to the user device. During the model application phase, the drone can return the acquired point cloud data to the user device in real time, allowing the user device to perform object recognition on the point cloud data using the trained model and return the object recognition results to the drone. Alternatively, after acquiring the model, the user device can deploy the model on the drone, allowing the drone to perform object recognition in real time during flight.

[0155] Please see Figure 4 This application provides a schematic diagram of a system architecture 400. For example... Figure 4 As shown, in this system architecture 400, the execution device 410 can be implemented by one or more servers, optionally in conjunction with other computing devices, such as data storage, routers, load balancers, etc. The execution device 410 can be deployed on a single physical site or distributed across multiple physical sites. The execution device 410 can use data from the data storage system 420 or call program code from the data storage system 420 to implement the point cloud data processing method provided in this application embodiment, thereby obtaining a model.

[0156] Users can interact with execution device 410 by operating their respective user devices (e.g., local device 401 and local device 402). Each local device can represent any computing device, such as a robot, smart car, personal computer, computer workstation, smartphone, tablet, smart camera, etc.

[0157] Each user's local device can interact with the execution device 410 through a communication network of any communication mechanism / standard. The communication network can be a wide area network, a local area network, a point-to-point connection, or any combination thereof.

[0158] In one implementation, the execution device 410 is used to implement the point cloud data processing method provided in the embodiments of this application, and sends the obtained model to the local device 401 and the local device 402 through a communication network, so that the local device 401 and the local device 402 can deploy and run the model.

[0159] In another implementation, one or more aspects of the execution device 410 may be implemented by each local device. For example, the local device 401 may provide local data or feedback calculation results to the execution device 410, or execute the point cloud data processing method provided in the embodiments of this application.

[0160] It should be noted that all the functions of the execution device 410 can also be implemented by a local device. For example, local device 401 implements the functions of the execution device 410 and provides services to its own users, or provides services to the users of local device 402.

[0161] In summary, the point cloud data processing method provided in the embodiments of this application can be applied to electronic devices, such as the execution device 410, local device 401 or local device 402 described above.

[0162] Please refer to Figure 5 , Figure 5 This is a schematic diagram of the structure of an electronic device 101 provided in an embodiment of this application. Figure 5 As shown, electronic device 101 includes processor 103, which is coupled to system bus 105. Processor 103 can be one or more processors, each of which can include one or more processor cores. A video adapter 107 drives a display 109, which is coupled to system bus 105. System bus 105 is coupled to input / output (I / O) bus via bus bridge 111. I / O interface 115 is coupled to the I / O bus. I / O interface 115 communicates with various I / O devices, such as input device 117 (e.g., touchscreen), external storage 121 (e.g., hard disk, floppy disk, optical disk, or USB flash drive), multimedia interface, etc. A transceiver 123 (capable of sending and / or receiving radio communication signals), a camera 155 (capable of capturing still and moving digital video images), and an external USB port 125. Optionally, the interface connected to I / O interface 115 can be a USB interface.

[0163] The processor 103 can be any conventional processor, including a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, or a combination thereof. Optionally, the processor can be a special-purpose device such as an ASIC.

[0164] Electronic device 101 can communicate with software deployment server 149 via network interface 129. Exemplarily, network interface 129 is a hardware network interface, such as a network interface card (NIC). Network 127 can be an external network, such as the Internet, or an internal network, such as Ethernet or a virtual private network (VPN). Optionally, network 127 can also be a wireless network, such as a WiFi network or a cellular network.

[0165] Hard disk drive interface 131 is coupled to system bus 105. Hardware driver interface is connected to hard disk drive 133. Internal memory 135 is coupled to system bus 105. Data running in internal memory 135 may include operating system (OS) 137, applications 143, and schedules of electronic device 101.

[0166] An operating system consists of the Shell 139 and the kernel 141. The Shell 139 is an interface between the user and the operating system kernel. The shell is the outermost layer of the operating system. The shell manages the interaction between the user and the operating system: waiting for user input, interpreting user input for the operating system, and processing various operating system outputs.

[0167] Kernel 141 consists of the parts of the operating system used to manage memory, files, peripherals, and system resources. Kernel 141 interacts directly with the hardware. The operating system kernel typically runs processes and provides inter-process communication, CPU time-slice management, interrupts, memory management, I / O management, and so on.

[0168] Please see Figure 6 , Figure 6 This is a flowchart illustrating a point cloud data processing method provided in an embodiment of this application. Figure 6 As shown, the method for processing point cloud data includes the following steps 601-605.

[0169] Step 601: Obtain the first point cloud data and the first image. The first point cloud data and the first image are obtained by collecting data from the first object.

[0170] In this embodiment, the first point cloud data and the first image can be obtained by collecting the same object (i.e., the first object) at the same point in time. That is, both the first point cloud data and the first image include the first object. In addition, besides the first object, the first point cloud data and the first image may also include other objects, that is, the first point cloud data and the first image may include multiple objects.

[0171] For example, in an autonomous driving scenario, both the first point cloud data and the first image can be collected by the autonomous vehicle at the same time. The first point cloud data and the first image may include a vehicle in front of the autonomous vehicle (i.e., the first object), and may also include obstacles such as pedestrians, water-filled barriers, or railings near the autonomous vehicle.

[0172] For example, the first point cloud data may be acquired by a LiDAR, and the first image may be acquired by an image sensor, with the LiDAR and image sensor deployed on the same device, such as an autonomous vehicle. In an autonomous vehicle, both the LiDAR and the image sensor may be used to acquire objects in the same direction, so that the acquired first point cloud data and the first image can include the same objects.

[0173] Alternatively, the first point cloud data and the first image can be acquired using the same depth camera, which may be deployed on a robot, for example. In this case, the depth camera, while acquiring the first image, can also acquire the spatial distance between each pixel in the first image and the depth camera. Therefore, based on the spatial distance between each pixel in the first image and the depth camera, each pixel in the first image can be converted into a point in the point cloud, thus obtaining the first point cloud data.

[0174] Step 602: Obtain the image block of the first object in the first image and the text name corresponding to the first object.

[0175] Understandably, since the first image typically includes other objects besides the first object, this step can obtain the image patch of the first object within the first image to locate its actual position. Furthermore, while determining the image patch of the first object in the first image, the text name corresponding to the first object can also be determined. Simply put, this step essentially aligns the image and text, determining the textual semantics of the image patch in the first image.

[0176] For example, in this step, the second model can be used to perform target recognition on the first image to obtain the image patch of the first object in the first image and the text name corresponding to the first object. That is, the second model can identify the objects included in the first image, mark the image patch of the identified first object in the first image, and give the text name corresponding to the first object. Here, the second model is a pre-trained model used to perform target recognition on the image. The second model can specifically be a convolutional neural network structure, but this embodiment does not specifically limit it.

[0177] For example, please see Figure 7, Figure 7 This is a schematic diagram illustrating how a second model can be used to identify objects in an image, as provided in an embodiment of this application. Figure 7 As shown, when the first image is input into a pre-trained second model, the second model can output an image patch from the first image (i.e., an image patch containing a car) and the corresponding text name "car". In addition to cars, the first image also includes objects such as trees, roadblocks, streetlights, and fire hydrants. Therefore, if the second model can recognize these objects, it can also output image patches corresponding to trees, roadblocks, streetlights, and fire hydrants, along with their corresponding text names.

[0178] Step 603: Based on the mapping relationship between the first point cloud data and the first image, and the image blocks, determine the first point cloud cluster corresponding to the first object in the first point cloud data.

[0179] Since the first point cloud data and the first image were acquired from the same object at the same time point, there is a mapping relationship between them. This mapping relationship enables the conversion between points in the first point cloud data and pixels in the first image. In other words, based on this mapping relationship, it is possible to determine the corresponding pixels in the first image for points in the first point cloud data, and vice versa.

[0180] Since both the first point cloud data and the first image may include multiple objects, after determining the image patch in the first image corresponding to the first object, the points corresponding to that image patch in the first point cloud data can be determined based on the mapping relationship, thereby obtaining the first point cloud cluster. The first point cloud cluster includes multiple points, and the points included in the first point cloud cluster are a subset of the points in the first point cloud data; that is, the first point cloud cluster is a subset of the first point cloud data.

[0181] Specifically, this mapping relationship can be obtained based on the device for acquiring the first point cloud data and the device for acquiring the first image. That is, the device for acquiring the first point cloud data and the device for acquiring the first image affect the conversion relationship between the first point cloud data and the first image.

[0182] For example, if the first point cloud data is acquired using a LiDAR and the first image is acquired using an image sensor, there will be a coordinate system transformation matrix between the LiDAR and the image sensor. This transformation matrix is ​​used to convert between the two-dimensional coordinate system of the image and the three-dimensional coordinate system of the point cloud data. Thus, based on the coordinate system transformation matrix between the LiDAR and the image sensor, the second point cloud cluster corresponding to the image patch in the first point cloud data can be determined. Then, clustering the points in the second point cloud cluster yields the first point cloud cluster, where the second point cloud cluster includes the first point cloud cluster; that is, the first point cloud cluster is a subset of the second point cloud cluster. The clustering of points in the second point cloud cluster can be performed using the DBSCAN clustering algorithm, which can extract points belonging to foreground objects within the second point cloud cluster.

[0183] It is understandable that, when the image patch corresponding to the first object is determined, this image patch may include other objects besides the first object; that is, the first object does not occupy the entire image patch. In this case, when the second point cloud cluster corresponding to the image patch in the first point cloud data is determined based on the coordinate system transformation matrix, this second point cloud cluster will also include other objects besides the first object.

[0184] Therefore, this embodiment can effectively extract the points belonging to the foreground object in the second point cloud cluster by performing clustering processing on the points in the second point cloud cluster, that is, extract the points belonging to the first object, thereby removing the points that do not belong to the first object and ensuring the accuracy of the final obtained first point cloud cluster.

[0185] For example, if both the first point cloud data and the first image are acquired by a depth camera, the depth camera will also have a coordinate transformation matrix, which is used to indicate the transformation relationship between the acquired first point cloud data and the first image.

[0186] Similarly, since an image patch may contain other objects besides the first object, the foreground portion of the image patch can be extracted first to obtain a foreground image patch containing only the first object. Then, based on the coordinate transformation matrix of the depth camera, the first point cloud cluster corresponding to the foreground image patch in the first point cloud data is determined. The foreground portion of the image patch can be extracted, for example, using the GrabCut segmentation algorithm.

[0187] In this scheme, by first extracting the foreground part of the image and then converting the foreground part of the image into a point cloud cluster, other objects besides the first object can be effectively extracted before the image is converted into a point cloud cluster, ensuring the accuracy of the final first point cloud cluster.

[0188] Step 604: Extract features of the first point cloud cluster and the text name through the first model to obtain the first point cloud features and the first text features. The first model includes a first network and a second network. The first point cloud features are obtained by the first network extracting features from the first point cloud cluster, and the first text features are obtained by the second network extracting features from the text name.

[0189] In this embodiment, the first network and the second network are structurally different and independent neural networks. The input to the first network is a first point cloud cluster, and the output of the first network is the first point cloud feature corresponding to the first point cloud cluster. The input to the second network is the text name of the first object, and the output of the second network is the first text feature corresponding to that text name.

[0190] For example, the first network can be an MLP, and the second network can be an attention network (such as a Transformer).

[0191] Step 605: Update the first model based on the first loss function to obtain the second model, wherein the first loss function is related to the first difference between the first point cloud features and the second text features.

[0192] In this embodiment, updating the first model based on the first loss function can specifically involve updating the parameters of the first network and the second network in the first model.

[0193] After extracting the first point cloud features and the first text features using the first model, a first loss function can be constructed based on these features. This first loss function can be positively correlated with the first difference between the first point cloud features and the second text features. That is, the larger the first difference between the first point cloud features and the first text features, the larger the first loss function; conversely, the smaller the first difference, the smaller the first loss function.

[0194] In this way, during the process of updating the first model based on the first loss function, the update objective of the first model is to minimize the first loss function, that is, to reduce the difference between the first point cloud features and the first text features corresponding to the same object. Training the first model based on the above steps allows it to learn to bring the point cloud features and text features corresponding to the same object as close as possible during training. This ensures that the trained second model, when extracting point cloud features and text features corresponding to the same object, can output point cloud features and text features with the smallest possible differences.

[0195] In this embodiment, based on point cloud data and images collected from the same object, the objects included in the image are first identified. Then, through the mapping relationship between the point cloud data and the image, the point cloud clusters corresponding to the objects in the image are determined. Thus, during the model training phase, the model can extract features from the point cloud clusters corresponding to the same object and the text, enabling comparative learning training of point cloud data features and text features. This ensures that subsequent object recognition in point cloud data can be achieved based on the trained model.

[0196] Since this solution uses images as an intermediary between point cloud data and text, the text corresponding to the point cloud data is determined in advance through the mapping relationship between point cloud data and images during the acquisition stage, thereby achieving alignment between point cloud data and text. This avoids the loss of the original data structure caused by projecting point cloud data into a depth map, which is beneficial for understanding objects in 3D scenes and can effectively improve the performance of the model.

[0197] The above describes the construction of a first loss function based on the first difference, enabling the first model to learn to narrow down point cloud features and text features corresponding to the same object during training. In some embodiments, the first loss function can be constructed in other ways, allowing the first model to simultaneously learn to narrow down point cloud features and text features corresponding to different objects while learning to narrow down point cloud features and text features corresponding to the same object.

[0198] For example, based on steps 601-605 above, the point cloud data processing method may further include: obtaining second point cloud features and second text features. The second point cloud features are obtained by the first network extracting features from the point cloud clusters corresponding to the second object, and the second text features are obtained by the second network extracting features from the text name of the second object. The first object and the second object are different objects. For example, the first object and the second object can be different objects belonging to the same first image; the second object can also be an object in other images, such as objects in other images belonging to the same batch of training data as the first image.

[0199] Furthermore, in step 605, the first loss function used to train the first model is correlated with a first difference and a second difference, where the second difference is the difference between the first text features and the second point cloud features. Additionally, the first loss function is positively correlated with the first difference, and negatively correlated with the second difference.

[0200] Since the first text feature corresponds to the first object, and the second point cloud feature corresponds to the second object, the second difference between the first text feature and the second point cloud feature is actually the difference between text features and point cloud features corresponding to different objects. Simultaneously, constructing a first loss function based on the first and second differences allows the first model to learn, during training, to both bring point cloud features and text features corresponding to the same object closer together and to learn to distance point cloud features and text features corresponding to different objects further apart. In this way, after training the second model, when extracting point cloud features and text features corresponding to the same object, the second model can output point cloud features and text features with the smallest possible differences; when extracting point cloud features and text features corresponding to different objects, the second model can output point cloud features and text features with the largest possible differences, thus enabling the second model to achieve better performance when identifying the category of point cloud data.

[0201] In one possible embodiment, the first model may further include a third network. This third network is used to extract features from the image; that is, the input to the third network is the image, and the output is the image's features. The third network is a structurally different and independent neural network from the first and second networks described above. This third network could be, for example, an attention network (such as a Transformer).

[0202] In the above method, after obtaining the image patch, features of the image patch can be extracted through a third network to obtain the first image features. Furthermore, in step 605, the first loss function is related to the first difference and the third difference, where the third difference is the difference between the first image features and the first point cloud features. Moreover, there is a positive correlation between the first loss function and the third difference. That is, the larger the third difference between the first point cloud features and the first image features, the larger the first loss function; conversely, the smaller the third difference between the first point cloud features and the first image features, the smaller the first loss function.

[0203] In this scheme, the loss function is constructed by introducing the differences between point cloud features and text features, as well as between point cloud features and image features. This enables the model to learn to approximate point cloud features, text features, and image features corresponding to the same object as much as possible during the training process based on the loss function, which can effectively improve the performance of the finally trained model.

[0204] Optionally, the first loss function can also be related to the first difference, second difference, and third difference mentioned above.

[0205] The above describes the process of training a model. The following will describe how to apply the trained model.

[0206] For example, after training the second model, second point cloud data can be acquired, which is the point cloud data for object recognition to be performed. For instance, the second point cloud data may be the point cloud data to be recognized collected by an autonomous vehicle during driving; or, the second point cloud data may be the point cloud data to be recognized collected by a robot during movement.

[0207] Then, the second point cloud data is input into the first network in the second model to obtain the third point cloud features extracted by the first network.

[0208] Secondly, multiple texts are input into the second network of the second model to obtain multiple text features extracted by the second network. Each text feature corresponds one-to-one with a specific text. Furthermore, these multiple texts can be determined based on the acquisition scenario of the second point cloud data. For example, in an autonomous driving scenario, the multiple texts could include words such as "car," "truck," "bicycle," "pedestrian," "roadblock," and "tree." In an indoor robot scenario, the multiple texts could include words such as "table," "chair," "sofa," "bed," "trash can," and "door."

[0209] Finally, the differences between the third point cloud feature and each of the multiple text features are determined, and based on these differences, the target text corresponding to the second point cloud data is determined. The target text corresponds to the target text feature, which is the text feature with the smallest difference from the third point cloud feature among the multiple text features.

[0210] In other words, after determining the difference between the third point cloud feature and each of the multiple text features, the differences between the third point cloud feature and each text feature can be sorted in ascending order, and the text feature with the smallest difference from the third point cloud feature (i.e., the target text feature mentioned above) can be determined. Then, the text corresponding to this text feature is taken as the target text.

[0211] The above describes the execution flow of the point cloud data processing method provided in the embodiments of this application. The following will describe in detail the execution process of the point cloud data processing method in actual application with specific examples.

[0212] For example, please refer to Figure 8 , Figure 8 This is a schematic diagram of a system architecture provided for an embodiment of this application. (For example...) Figure 8 As shown, the system architecture includes a sensor data collection module, a general perception module, and a decision reasoning module.

[0213] The sensor data collection module is used to collect point cloud data and images and transmit them to the general perception module.

[0214] The general perception module is used to realize object recognition in open worlds, such as obstacle recognition or general object recognition in outdoor or indoor environments.

[0215] The decision reasoning module performs corresponding decision reasoning based on the object recognition results transmitted by the general perception module.

[0216] For example, please refer to Figure 9A and Figure 9B , Figure 9A This application provides an embodiment of object recognition in an outdoor open world. Figure 9B This is a schematic diagram illustrating object recognition in an indoor open world, as provided in an embodiment of this application. Figure 9A As shown, the aforementioned general perception module can be applied to the in-vehicle system of autonomous vehicles to assist autonomous vehicles in perceiving any type of obstacle in the open world. This enables the decision-making and reasoning module of autonomous vehicles to perform obstacle avoidance and route planning control based on different object categories, thereby achieving safe driving.

[0217] like Figure 9B As shown, the aforementioned general perception module can be applied to robots to assist them in locating and recognizing any object in an open-world environment, thereby enabling the robot's decision-making and reasoning module to execute corresponding obstacle avoidance or cleaning commands.

[0218] In addition, the aforementioned general perception module also includes a trimodal data alignment module, a trimodal contrast pre-trained three-tower model, and a multimodal joint discrimination module. Here, trimodal refers to three different data modalities: text, image, and point cloud data.

[0219] The three-modal alignment module is used to align point cloud data and image data based on the coordinate system transformation matrix of the sensor that acquired the point cloud data and image. Furthermore, it aligns text and images using a pre-trained image-text model, thereby achieving the alignment of point cloud data, image, and text data through the image as an intermediary.

[0220] The trimodal contrastive pre-trained three-tower model is used to perform contrastive pre-training based on point cloud data, images, and text corresponding to the same object, bringing the feature spaces of the three modalities corresponding to the same object closer together, so that the three feature extractors can output aligned and correlated features.

[0221] The multimodal joint discrimination module combines image features and point cloud features, utilizes the feature relationship with text, calculates the association with open domain text, and obtains the classification result of general category objects.

[0222] For example, please refer to Figure 10 , Figure 10 This is a schematic diagram illustrating a process for locating and recognizing 3D objects in an open world, provided as an embodiment of this application. Figure 10 As shown, the process for locating and recognizing 3D objects in an open world includes the following steps S1-S3.

[0223] Step S1: Align the three-modal data.

[0224] (1) Align text and images

[0225] In outdoor autonomous driving scenarios, this embodiment uses point cloud data P collected by LiDAR. s Scene images acquired by image sensors I s The input is a scene s∈|S|, used to locate and classify general category 3D objects.

[0226] For example, please refer to Figure 11 , Figure 11 This is a schematic diagram illustrating a process for trimodal data alignment in an outdoor scene, as provided in an embodiment of this application. Figure 11 As shown, using the open-domain vocabulary X T and 2D image I s (i.e., the first image mentioned above) is used as input, and the pre-trained image-text detection model M is used. VLM (i.e., the second model mentioned above), to obtain the open-domain vocabulary X T The object corresponding to the text in the 2D image I s Image region set on Right now:

[0227]

[0228] Each 2D image block Corresponding to an open text vocabulary

[0229] (2) Aligning image and point cloud features

[0230] like Figure 11 As shown, the coordinate system transformation matrix G between the lidar and the image sensor is used. OUT Obtaining 2D image patches Corresponding 3D point cloud cone clusters Specifically, the process of projecting 3D point clouds into 2D image pixels is as follows:

[0231] [u,v]=G OUT ×[x,y,z]

[0232]

[0233] Where [u,v] are the coordinates of pixels in the 2D image, [x,y,z] are the coordinates of points in the 3D point cloud data, I is the intrinsic parameter matrix of the image sensor, and R... C R is the extrinsic parameter matrix of the image sensor. L The extrinsic parameter matrix of the lidar is used, therefore the 2D image block contains... The 3D point cloud corresponding to the projected pixel [u,v]′ is the obtained point cloud cluster. Then, the point cloud clusters were obtained using the DBSCAN clustering algorithm. 3D objects inside This allows for the removal of background objects from point cloud clusters.

[0234] Therefore, the final data set of text-image-point cloud triplet can be obtained.

[0235] The above describes the process of aligning images, point cloud data, and text acquired in outdoor scenes. In indoor scenes, the methods for acquiring point cloud data and images differ, therefore the process of aligning images and point cloud data also differs. However, the process of aligning images and text is the same for both indoor and outdoor scenes.

[0236] For example, please refer to Figure 12 , Figure 12 This is a schematic diagram illustrating a process for trimodal data alignment in an indoor scene, as provided in an embodiment of this application. Figure 12 As shown, in indoor scenes, both the images and point cloud data are acquired by a depth camera.

[0237] After aligning the image and text, the GrabCut segmentation algorithm is first used to extract 2D object image patches. Foreground part Using the coordinate transformation matrix G of the depth camera IN Obtain image region Corresponding 3D object point cloud clusters Specifically, the coordinates (x, y, z) of each point in the point cloud cluster can be obtained by projecting the positions (u′, v′, d′) of the pixels in the image, as shown below:

[0238]

[0239] G IN =I×R C

[0240] Where [u′,v′,d′] are the coordinates and depth of a pixel in the 2D image, [x,y,z] are the coordinates of a point in the 3D point cloud data, I is the intrinsic parameter matrix of the camera, and R C is the extrinsic parameter matrix of the camera.

[0241] Therefore, the final data set of text-image-point cloud triplet can be obtained.

[0242] Step S2, trimodal feature pre-training.

[0243] After aligning the text, image, and point cloud data, a trimodal contrastive pre-trained tri-tower model can be used to perform contrastive pre-training on the text, image, and point cloud data. See also... Figure 13 , Figure 13 This is a schematic diagram illustrating a process for performing contrastive pre-training on text, image, and point cloud data using a three-modal contrastive pre-training three-tower model, as provided in an embodiment of this application.

[0244] like Figure 13 As shown, the trimodal contrastive pre-trained three-tower model (i.e., the first model mentioned above) includes a point cloud feature extractor. (i.e., the first network mentioned above), image feature extractor (i.e., the third network mentioned above) and text feature extractor (i.e., the second network mentioned above). Based on the text-image-point cloud triplet data set D obtained in step S1, the aligned point cloud data, image, and text are input into three feature extractors respectively to obtain the corresponding 3D point cloud features f. P 2D image features f I and text features f T Then, by comparing the pre-training methods, the feature spaces of the three modalities are brought closer together, so that the three feature extractors can output aligned and correlated features. Specifically, the loss function constructed during the comparison pre-training process is as follows:

[0245]

[0246]

[0247]

[0248] Where L(T,I,P) is the loss function; N represents the number of text-image-point cloud triplet data pairs; and τ is a temperature coefficient used to adjust the intensity of contrastive pre-training. The goal of performing contrastive pre-training based on the above loss function is to: bring the features between text and point clouds corresponding to the same object and the features between images and point clouds corresponding to the same object closer together, and to widen the features between text and point clouds corresponding to different objects and the features between images and point clouds corresponding to different objects further apart.

[0249] In this way, by training the trimodal contrastive pre-trained three-tower model in the above manner, the trained trimodal contrastive pre-trained three-tower model can output similar features when processing text, image and point cloud data corresponding to the same object, and output different features when processing text, image and point cloud data corresponding to different objects.

[0250] Step S3, Three-modal joint feature discrimination

[0251] Step S3 is performed during the testing phase (i.e., the application phase of the model), based on the trained trimodal comparison pre-trained three-tower model. Please refer to [link / reference]. Figure 14 , Figure 14 This is a schematic diagram of a three-modal joint feature discrimination process provided in an embodiment of this application.

[0252] like Figure 14 As shown, in practical application scenarios, images of the open world are first acquired through sensors. and dotted cloud clusters and the image and dotted cloud clusters Input to image feature extractor respectively Point cloud feature extractor Obtain the corresponding image features and point cloud features

[0253] Then, the vocabulary from the text library is input into the text feature extractor. Obtaining text features Where K represents the length of the text library (i.e. the number of texts), and C represents the feature dimension (i.e., the feature vector of each text is C-dimensional).

[0254] Secondly, the sum of image features and point cloud features is calculated, and the feature distance between the sum of image features and point cloud features and text features is calculated. The text feature with the shortest distance is selected as the classification result with the highest probability.

[0255]

[0256] in, This represents the feature distance between the sum of image features and point cloud features and the text features; This represents the classification probability corresponding to each text.

[0257] The methods provided in the embodiments of this application have been described in detail above. Next, the device for performing the above methods provided in the embodiments of this application will be described.

[0258] Please see Figure 15 , Figure 15This is a schematic diagram of a point cloud data processing device provided in an embodiment of this application.

[0259] like Figure 15 As shown, the point cloud data processing device includes:

[0260] The acquisition module 1501 acquires the first point cloud data and the first image, which are obtained by collecting data from the first object.

[0261] The acquisition module 1501 is also used to acquire the image block of the first object in the first image and the text name corresponding to the first object;

[0262] Processing module 1502 is used to determine the first point cloud cluster corresponding to the first object in the first point cloud data based on the mapping relationship between the first point cloud data and the first image and the image blocks;

[0263] The processing module 1502 is also used to extract features of the first point cloud cluster and the text name through the first model to obtain the first point cloud features and the first text features. The first model includes a first network and a second network. The first point cloud features are obtained by the first network extracting features from the first point cloud cluster, and the first text features are obtained by the second network extracting features from the text name.

[0264] The processing module 1502 is also used to update the first model based on the first loss function to obtain the second model, wherein the first loss function is related to the first difference between the first point cloud features and the second text features.

[0265] In one possible implementation, the acquisition module 1501 is further used to acquire second point cloud features and second text features. The second point cloud features are obtained by the first network extracting features from the point cloud clusters corresponding to the second object. The second text features are obtained by the second network extracting features from the text name of the second object. The first object and the second object are different objects.

[0266] The first loss function is related to the first difference and the second difference, where the second difference is the difference between the first text feature and the second point cloud feature.

[0267] In one possible implementation, the first loss function is positively correlated with the first difference, and the first loss function is negatively correlated with the second difference.

[0268] In one possible implementation, the first model also includes a third network;

[0269] Processing module 1502 is also used for:

[0270] The first image features are obtained by extracting features from image patches using a third network;

[0271] The first loss function is related to the first difference and the third difference, where the third difference is the difference between the first image features and the first point cloud features.

[0272] In one possible implementation, the acquisition module 1501 is also used to acquire second point cloud data;

[0273] The processing module 1502 is also used to input the second point cloud data into the first network in the second model to obtain the third point cloud features;

[0274] Processing module 1502 is also used to input multiple texts into the second network in the second model to obtain multiple text features;

[0275] The processing module 1502 is also used to determine the target text corresponding to the second point cloud data based on the difference between the third point cloud feature and each of the multiple text features, wherein the target text corresponds to the target text feature, and the target text feature is the text feature with the smallest difference from the third point cloud feature among the multiple text features.

[0276] In one possible implementation, the mapping relationship is obtained based on the device for acquiring the first point cloud data and the device for acquiring the first image.

[0277] In one possible implementation, the first point cloud data is acquired by LiDAR, the first image is acquired by an image sensor, and the LiDAR and the image sensor are deployed on the same device.

[0278] In one possible implementation, the processing module 1502 is further configured to:

[0279] Based on the coordinate system transformation matrix between the lidar and the image sensor, the second point cloud cluster corresponding to the image patch in the first point cloud data is determined.

[0280] Clustering is performed on the points in the second point cloud cluster to obtain the first point cloud cluster, where the second point cloud cluster includes the first point cloud cluster.

[0281] In one possible implementation, the first point cloud data and the first image are acquired using the same depth camera.

[0282] In one possible implementation, the processing module 1502 is further configured to:

[0283] Extract the foreground portion from the image patch to obtain the foreground image patch;

[0284] Based on the coordinate system transformation matrix of the depth camera, the first point cloud cluster corresponding to the foreground image patch in the first point cloud data is determined.

[0285] In one possible implementation, the processing module 1502 is further configured to:

[0286] The second model is used to perform target recognition on the first image to obtain the image patch of the first object in the first image and the text name corresponding to the first object. The second model is a pre-trained model.

[0287] Please see Figure 16 , Figure 16 This is a schematic diagram of an execution device provided in an embodiment of this application. The execution device 1600 can specifically be a mobile phone, tablet, laptop, smart wearable device, server, etc., and is not limited thereto. Specifically, the execution device 1600 includes: a receiver 1601, a transmitter 1602, a processor 1603, and a memory 1604 (wherein the execution device 1600 may have one or more processors 1603). Figure 16 (Taking a processor as an example), processor 1603 may include application processor 16031 and communication processor 16032. In some embodiments of this application, receiver 1601, transmitter 1602, processor 1603 and memory 1604 may be connected via bus or other means.

[0288] Memory 1604 may include read-only memory and random access memory, and provides instructions and data to processor 1603. A portion of memory 1604 may also include non-volatile random access memory (NVRAM). Memory 1604 stores processor and operation instructions, executable modules, or data structures, or subsets thereof, or extended sets thereof, wherein the operation instructions may include various operation instructions for implementing various operations.

[0289] Processor 1603 controls the operation of the execution device. In specific applications, the various components of the execution device are coupled together through a bus system, which may include not only the data bus, but also power buses, control buses, and status signal buses. However, for clarity, all buses are referred to as the bus system in the diagram.

[0290] The methods disclosed in the embodiments of this application described above can be applied to processor 1603, or implemented by processor 1603. Processor 1603 can be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by integrated logic circuits in the hardware of processor 1603 or by instructions in software form. The processor 1603 described above can be a general-purpose processor, a digital signal processor (DSP), a microprocessor, or a microcontroller, and may further include application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0291] The processor 1603 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory 1604. The processor 1603 reads information from memory 1604 and, in conjunction with its hardware, completes the steps of the above methods.

[0292] Receiver 1601 can be used to receive input digital or character information, and to generate signal inputs related to the settings and function control of the execution device. Transmitter 1602 can be used to output digital or character information through the first interface; transmitter 1602 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group; transmitter 1602 may also include a display device such as a display screen.

[0293] The electronic device provided in this application embodiment can specifically be a chip, which includes a processing unit and a communication unit. The processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, pins, or circuits. The processing unit can execute computer execution instructions stored in the storage unit to cause the chip in the execution device to execute the model structure determination method described in the above embodiments, or to cause the chip in the training device to execute the model structure determination method described in the above embodiments. Optionally, the storage unit can be an internal storage unit within the chip, such as a register or cache. Alternatively, the storage unit can be an external storage unit located within the wireless access device, such as a read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, such as random access memory (RAM).

[0294] For details, please refer to Figure 17 , Figure 17 This is a schematic diagram of a chip provided in an embodiment of this application. The chip can be represented as a neural network processor (NPU) 1700. The NPU 1700 is mounted as a coprocessor on the host CPU, and tasks are assigned by the host CPU. The core part of the NPU is the arithmetic circuit 1703, which is controlled by the controller 1704 to extract matrix data from the memory and perform multiplication operations.

[0295] In some implementations, the arithmetic circuit 1703 internally includes multiple processing engines (PEs). In some implementations, the arithmetic circuit 1703 is a two-dimensional pulsating array. The arithmetic circuit 1703 can also be a one-dimensional pulsating array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 1703 is a general-purpose matrix processor.

[0296] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from the weight memory 1702 and caches it in each PE of the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from the input memory 1701 and performs matrix operations with matrix B. The partial result or the final result of the obtained matrix is ​​stored in the accumulator 1708.

[0297] Unified memory 1706 is used to store input and output data. Weight data is directly transferred to weight memory 1702 via Direct Memory Access Controller (DMAC) 1705. Input data is also transferred to unified memory 1706 via DMAC.

[0298] BIU stands for Bus Interface Unit, which is used for interaction between the AXI bus and the DMAC and the Instruction Fetch Buffer (IFB) 1709.

[0299] The Bus Interface Unit (BIU) 1710 is used by the instruction fetch memory 1709 to fetch instructions from external memory, and also by the memory access controller 1705 to fetch the original data of the input matrix A or the weight matrix B from external memory.

[0300] The DMAC is mainly used to move input data from external memory DDR to unified memory 1706, or to weight data to weight memory 1702, or to input data to input memory 1701.

[0301] The vector computation unit 1707 includes multiple arithmetic processing units that, when needed, further process the output of the computation circuit 1703, such as vector multiplication, vector addition, exponential operations, logarithmic operations, size comparisons, etc. It is mainly used for computation in non-convolutional / fully connected layers of neural networks, such as batch normalization, pixel-level summation, and upsampling of feature planes.

[0302] In some implementations, the vector computation unit 1707 can store the processed output vector in the unified memory 1706. For example, the vector computation unit 1707 can apply a linear function, or a nonlinear function, to the output of the computation circuit 1703, such as performing linear interpolation on feature planes extracted by a convolutional layer, or, for example, accumulating a vector of values ​​to generate activation values. In some implementations, the vector computation unit 1707 generates normalized values, pixel-level summed values, or both. In some implementations, the processed output vector can be used as an activation input to the computation circuit 1703, for example, for use in subsequent layers of the neural network.

[0303] The instruction fetch buffer 1709 connected to the controller 1704 is used to store the instructions used by the controller 1704.

[0304] Unified memory 1706, input memory 1701, weighted memory 1702, and instruction fetch memory 1709 are all on-chip memories. External memory is proprietary to this NPU hardware architecture.

[0305] The processor mentioned above can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits used to control the execution of the above program.

[0306] Please refer to Figure 18 , Figure 18 This is a schematic diagram of a computer-readable storage medium provided in an embodiment of this application. This application also provides a computer-readable storage medium in some embodiments, wherein the above-described... Figure 6 The disclosed method can be implemented as computer program instructions encoded in a machine-readable format on a computer-readable storage medium or on other non-transitory media or articles of art.

[0307] Figure 18 A conceptual partial view of an example computer-readable storage medium arranged according to at least some of the embodiments shown herein is illustrated schematically. The example computer-readable storage medium includes a computer program for executing computer processes on a computing device.

[0308] In one embodiment, the computer-readable storage medium 1800 is provided using a signal bearer medium 1801. The signal bearer medium 1801 may include one or more program instructions 1802, which, when executed by one or more processors, can provide the above-mentioned... Figure 6 The described function or part of the function.

[0309] In some examples, the signal carrying medium 1801 may include a computer-readable medium 1803, such as, but not limited to, a hard disk drive, a compact disc (CD), a digital video optical disc (DVD), a digital magnetic tape, a memory, ROM, or RAM, etc.

[0310] In some embodiments, the signal-bearing medium 1801 may include a computer-recordable medium 1804, such as, but not limited to, a memory, a read / write (R / W) CD, a R / W DVD, etc. In some embodiments, the signal-bearing medium 1801 may include a communication medium 1805, such as, but not limited to, digital and / or analog communication media (e.g., fiber optic cables, waveguides, wired communication links, wireless communication links, etc.). Therefore, for example, the signal-bearing medium 1801 may be transmitted by a wireless communication medium 1805 (e.g., a wireless communication medium conforming to the IEEE 802.11 standard or other transmission protocols).

[0311] One or more program instructions 1802 may be, for example, computer-executable instructions or logical implementation instructions. In some examples, the computing device may be configured to provide various operations, functions, or actions in response to one or more program instructions 1802 conveyed to the computing device via a computer-readable medium 1803, a computer-recordable medium 1804, and / or a communication medium 1805.

[0312] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the accompanying drawings of the device embodiments provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0313] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods of the various embodiments of this application.

[0314] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0315] A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A method for processing point cloud data, characterized in that, include: Acquire first point cloud data and first image, wherein the first point cloud data and first image are obtained by collecting data from a first object; Obtain the image block of the first object in the first image and the text name corresponding to the first object; Based on the mapping relationship between the first point cloud data and the first image, and the image block, the first point cloud cluster corresponding to the first object in the first point cloud data is determined; The first point cloud cluster and the text name are extracted using a first model to obtain first point cloud features and first text features. The first model includes a first network and a second network. The first point cloud features are obtained by the first network extracting features from the first point cloud cluster, and the first text features are obtained by the second network extracting features from the text name. The first model is updated based on the first loss function to obtain the second model, wherein the first loss function is related to the first difference between the first point cloud features and the second text features, and the second model is used to realize the recognition of objects in the point cloud data.

2. The method according to claim 1, characterized in that, The method further includes: The second point cloud feature and the second text feature are obtained. The second point cloud feature is obtained by the first network extracting features from the point cloud cluster corresponding to the second object. The second text feature is obtained by the second network extracting features from the text name of the second object. The first object and the second object are different objects. The first loss function is related to the first difference and the second difference, where the second difference is the difference between the first text feature and the second point cloud feature.

3. The method according to claim 2, characterized in that, The first loss function is positively correlated with the first difference, and the first loss function is negatively correlated with the second difference.

4. The method according to any one of claims 1-3, characterized in that, The first model also includes a third network; The method further includes: The features of the image patch are extracted using the third network to obtain the first image features; The first loss function is related to the first difference and the third difference, wherein the third difference is the difference between the first image feature and the first point cloud feature.

5. The method according to any one of claims 1-3, characterized in that, The method further includes: Obtain the second point cloud data; The second point cloud data is input into the first network in the second model to obtain the third point cloud features; Multiple texts are input into the second network of the second model to obtain multiple text features; Based on the difference between the third point cloud feature and each of the plurality of text features, the target text corresponding to the second point cloud data is determined, wherein the target text corresponds to a target text feature, and the target text feature is the text feature with the smallest difference from the third point cloud feature among the plurality of text features.

6. The method according to any one of claims 1-3, characterized in that, The mapping relationship is obtained based on the device for acquiring the first point cloud data and the device for acquiring the first image.

7. The method according to any one of claims 1-3, characterized in that, The first point cloud data is acquired by LiDAR, and the first image is acquired by an image sensor, with the LiDAR and the image sensor deployed on the same device.

8. The method according to claim 7, characterized in that, The step of determining the first point cloud cluster corresponding to the first object in the first point cloud data based on the mapping relationship between the first point cloud data and the first image, and the image patch, includes: Based on the coordinate system transformation matrix between the lidar and the image sensor, the second point cloud cluster corresponding to the image patch in the first point cloud data is determined; Clustering is performed on the points in the second point cloud cluster to obtain the first point cloud cluster, wherein the second point cloud cluster includes the first point cloud cluster.

9. The method according to any one of claims 1-3, characterized in that, The first point cloud data and the first image were acquired using the same depth camera.

10. The method according to claim 9, characterized in that, The step of determining the first point cloud cluster corresponding to the first object in the first point cloud data based on the mapping relationship between the first point cloud data and the first image, and the image patch, includes: Extract the foreground portion from the image block to obtain the foreground image block; Based on the coordinate system transformation matrix of the depth camera, the first point cloud cluster corresponding to the foreground image patch in the first point cloud data is determined.

11. The method according to any one of claims 1-3, characterized in that, The step of obtaining the image patch of the first object in the first image and the text name corresponding to the first object includes: The first image is used to identify the target by a pre-trained model, and the image patch of the first object in the first image and the text name corresponding to the first object are obtained.

12. A point cloud data processing device, characterized in that, include: The acquisition module acquires first point cloud data and first image, which are obtained by collecting data from a first object. The acquisition module is further configured to acquire the image block of the first object in the first image and the text name corresponding to the first object; The processing module is used to determine the first point cloud cluster corresponding to the first object in the first point cloud data based on the mapping relationship between the first point cloud data and the first image and the image block; The processing module is further configured to extract features of the first point cloud cluster and the text name through the first model to obtain first point cloud features and first text features, wherein the first model includes a first network and a second network, the first point cloud features are obtained by the first network extracting features from the first point cloud cluster, and the first text features are obtained by the second network extracting features from the text name. The processing module is further configured to update the first model based on a first loss function to obtain a second model, wherein the first loss function is related to a first difference between the first point cloud features and the second text features, and the second model is configured to recognize objects in the point cloud data.

13. The apparatus according to claim 12, characterized in that, The acquisition module is further configured to acquire a second point cloud feature and a second text feature. The second point cloud feature is obtained by the first network extracting features from the point cloud cluster corresponding to the second object. The second text feature is obtained by the second network extracting features from the text name of the second object. The first object and the second object are different objects. The first loss function is related to the first difference and the second difference, where the second difference is the difference between the first text feature and the second point cloud feature.

14. The apparatus according to claim 13, characterized in that, The first loss function is positively correlated with the first difference, and the first loss function is negatively correlated with the second difference.

15. The apparatus according to any one of claims 12-14, characterized in that, The first model also includes a third network; The processing module is further configured to: The features of the image patch are extracted using the third network to obtain the first image features; The first loss function is related to the first difference and the third difference, wherein the third difference is the difference between the first image feature and the first point cloud feature.

16. The apparatus according to any one of claims 12-14, characterized in that, The acquisition module is also used to acquire second point cloud data; The processing module is also used to input the second point cloud data into the first network in the second model to obtain the third point cloud features; The processing module is also used to input multiple texts into the second network in the second model to obtain multiple text features; The processing module is further configured to determine the target text corresponding to the second point cloud data based on the difference between the third point cloud feature and each of the plurality of text features, wherein the target text corresponds to a target text feature, and the target text feature is the text feature with the smallest difference from the third point cloud feature among the plurality of text features.

17. The apparatus according to any one of claims 12-14, characterized in that, The mapping relationship is obtained based on the device for acquiring the first point cloud data and the device for acquiring the first image.

18. The apparatus according to any one of claims 12-14, characterized in that, The first point cloud data is acquired by LiDAR, and the first image is acquired by an image sensor, with the LiDAR and the image sensor deployed on the same device.

19. The apparatus according to claim 18, characterized in that, The processing module is further configured to: Based on the coordinate system transformation matrix between the lidar and the image sensor, the second point cloud cluster corresponding to the image patch in the first point cloud data is determined; Clustering is performed on the points in the second point cloud cluster to obtain the first point cloud cluster, wherein the second point cloud cluster includes the first point cloud cluster.

20. The apparatus according to any one of claims 12-14, characterized in that, The first point cloud data and the first image were acquired using the same depth camera.

21. The apparatus according to claim 20, characterized in that, The processing module is further configured to: Extract the foreground portion from the image block to obtain the foreground image block; Based on the coordinate system transformation matrix of the depth camera, the first point cloud cluster corresponding to the foreground image patch in the first point cloud data is determined.

22. The apparatus according to any one of claims 12-14, characterized in that, The processing module is further configured to: The first image is used to identify the target by a pre-trained model, and the image patch of the first object in the first image and the text name corresponding to the first object are obtained.

23. A point cloud data processing device, characterized in that, The device includes a memory and a processor; the memory stores code, and the processor is configured to execute the code, wherein when the code is executed, the device performs the method as described in any one of claims 1 to 11.

24. A computer storage medium, characterized in that, The computer storage medium stores instructions that, when executed by the computer, cause the computer to perform the method according to any one of claims 1 to 11.

25. A computer program product, characterized in that, The computer program product stores instructions that, when executed by a computer, cause the computer to perform the method described in any one of claims 1 to 11.

Images