Three-dimensional target identification method and device, electronic equipment, chip and medium

The three-dimensional semantic Gaussian is generated through two-dimensional backbone network and sparse convolution, and its physical properties are updated using the deformable attention mechanism. Combined with the Gaussian voxel sputtering algorithm, the problem of low three-dimensional target recognition performance is solved, achieving efficient and refined Three-dimensional object recognition, suitable for complex scenarios such as autonomous driving.

CN119068474APending Publication Date: 2024-12-03TSINGHUA UNIVERSITY
View PDF 0 Cites 6 Cited by

Patent Information

Application Number
CN202411056229.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-02
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

The existing three-dimensional target recognition technology has problems with low recognition performance and high algorithm complexity, especially in scenarios such as autonomous driving, which is difficult to effectively capture obstacles of any shape, affecting safety and robustness.

Method used

A two-dimensional backbone network is used to extract multi-view image features, generate interrelated three-dimensional semantic Gaussians through sparse convolution, and update the physical properties of three-dimensional semantic Gaussians using the deformable attention mechanism, and finally achieve efficient and fine use of the Gaussian voxel sputtering algorithm. three-dimensional target recognition.

Benefits of technology

It significantly improves the performance of three-dimensional object recognition, improves the ability to understand and express complex three-dimensional structures, enhances the ability to handle complex and changing scenarios, and improves the accuracy and robustness of feature representation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119068474A_ABST
    Figure CN119068474A_ABST
Patent Text Reader

Abstract

The invention provides a three-dimensional target recognition method and device, electronic equipment, a chip and a medium, and relates to the technical field of machine learning. The three-dimensional target identification method comprises the following steps: based on a multi-view image of a target object, extracting multi-view and multi-scale image features of the target object by using a two-dimensional backbone network; mutually associated three-dimensional semantic gausses are generated through sparse convolution, wherein the three-dimensional semantic gausses comprise the three-dimensional gausses of the target object features; based on the image features, updating the three-dimensional semantic Gaussian through a deformable attention mechanism, and adjusting the physical attributes of the three-dimensional semantic Gaussian; and according to the three-dimensional semantic Gaussian, identifying a target object through a Gaussian voxel sputtering algorithm. Through the technical scheme provided by the invention, the problem of low three-dimensional target recognition performance in related technologies is solved, and the three-dimensional object recognition performance is remarkably improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of machine learning, and in particular to a three-dimensional target recognition method, device, electronic device, chip and medium. Background Art

[0002] With the development of sensors and artificial intelligence technology, in order to capture objects of any shape, the recognition of two-dimensional objects has evolved to three-dimensional objects. In related technologies, a grid-based method is used to predict the occupancy state of each voxel in the surrounding three-dimensional space through three-dimensional object recognition to complete the recognition of three-dimensional objects. However, this method has the problem of low recognition performance. Summary of the invention

[0003] The present disclosure provides a three-dimensional target recognition method, device, electronic device, chip and medium to solve the problem of low three-dimensional target recognition performance in related technologies. A two-dimensional backbone network is used to process multi-view camera images to generate multi-view and multi-scale image features; then a Gaussian autoencoder module is used to complete the interaction between three-dimensional semantic Gaussians; next, image feature cross attention is used to update the three-dimensional semantic Gaussian features by the image features; then the physical properties of the three-dimensional semantic Gaussian are adjusted; finally, the Gaussian voxel sputtering module generates efficient and precise three-dimensional target recognition results from the three-dimensional semantic Gaussian.

[0004] A first aspect of the present disclosure provides a three-dimensional target recognition method, the method comprising:

[0005] Based on the multi-view images of the target object, a two-dimensional backbone network is used to extract the multi-view and multi-scale image features of the target object;

[0006] Generate mutually correlated three-dimensional semantic Gaussians through sparse convolution, where the three-dimensional semantic Gaussians include the three-dimensional Gaussians of the target object features;

[0007] Based on the image features, the 3D semantic Gaussian is updated through a deformable attention mechanism, and the physical properties of the 3D semantic Gaussian are adjusted;

[0008] According to the 3D semantic Gaussian, the target object is identified by Gaussian voxel sputtering algorithm.

[0009] In one embodiment of the present disclosure, generating mutually correlated three-dimensional semantic Gaussians through sparse convolution includes:

[0010] The center of the 3D semantic Gaussian is used as the spatial point to generate a 3D semantic point cloud;

[0011] voxelize the 3D semantic point cloud into 3D semantic voxels;

[0012] Based on the three-dimensional semantic voxels, three-dimensional sparse convolution is used to process the three-dimensional semantic voxels, and the associated three-dimensional semantic voxels correspond to three-dimensional semantic Gaussians.

[0013] In one embodiment of the present disclosure, based on image features, a three-dimensional semantic Gaussian is updated through a deformable attention mechanism, including:

[0014] Based on the camera parameters corresponding to the multi-view images, each 3D semantic Gaussian is projected into the pixel coordinate system through projection transformation to generate a 2D semantic image.

[0015] Based on each pixel of the two-dimensional semantic image, a deformable attention mechanism is used to generate the offset and attention weight of the target pixel in the pixel coordinate system to obtain a semantic bias feature map. The target pixel is any pixel in the two-dimensional semantic image.

[0016] Upsampling the semantic bias feature map by bilinear interpolation to obtain the features of the target reference point corresponding to the target three-dimensional semantic Gaussian in the semantic bias feature, wherein the target three-dimensional semantic Gaussian corresponds to the target pixel point, and each three-dimensional semantic Gaussian includes the target three-dimensional semantic Gaussian;

[0017] The features of the target reference points are weighted and summed according to the attention weights to obtain the features of the target 3D semantic Gaussian to update the 3D semantic Gaussian.

[0018] In one embodiment of the present disclosure, after updating the three-dimensional semantic Gaussian, the method further includes:

[0019] Adjust the physical properties of the 3D semantic Gaussian, including:

[0020] Based on the features of three-dimensional semantic Gaussian, a multi-layer perceptron is used to predict the residual value corresponding to the parameter value of the physical attribute, and the sum of the residual value and the parameter value is used to update the parameter value to optimize the three-dimensional semantic Gaussian. The physical attributes include covariance, color, opacity and spherical harmonic coefficients.

[0021] In one embodiment of the present disclosure, according to the three-dimensional semantic Gaussian, a target object is identified by a Gaussian voxel sputtering algorithm, including:

[0022] Obtaining three-dimensional semantic voxels;

[0023] According to the covariance of the three-dimensional semantic voxels, a distance threshold corresponding to the three-dimensional semantic Gaussian is determined;

[0024] If the distance between the center position of the 3D semantic Gaussian and the 3D semantic voxel is less than or equal to the distance threshold, the category of the 3D semantic voxel is used as the category label of the 3D semantic Gaussian, otherwise, the 3D semantic Gaussian is discarded;

[0025] The category label is used as the recognition result of the target object.

[0026] In one embodiment of the present disclosure, before generating mutually correlated three-dimensional semantic Gaussians through sparse convolution, the method further includes:

[0027] Generate at least two three-dimensional Gaussians that obey multivariate Gaussian distribution in three-dimensional space;

[0028] Initialize the 3D semantic Gaussian using the 3D Gaussian.

[0029] A second aspect of the present disclosure provides a three-dimensional object recognition device, the device comprising:

[0030] A feature extraction module is used to extract multi-view and multi-scale image features of a target object based on a multi-view image of the target object using a two-dimensional backbone network;

[0031] An association generation module, used for generating mutually associated three-dimensional semantic Gaussians through sparse convolution, where the three-dimensional semantic Gaussians include three-dimensional Gaussians of target object features;

[0032] A 3D semantic Gaussian update module is used to update the 3D semantic Gaussian through a deformable attention mechanism based on image features and adjust the physical properties of the 3D semantic Gaussian;

[0033] The target recognition module is used to recognize the target object through the Gaussian voxel sputtering algorithm based on the three-dimensional semantic Gaussian.

[0034] The third aspect embodiment of the present disclosure proposes an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any one of the methods in the first aspect embodiment of the present disclosure.

[0035] The fourth aspect embodiment of the present disclosure proposes a non-transitory computer-readable storage medium storing computer instructions, characterized in that the computer instructions are used to enable a computer to execute the method in the first aspect embodiment of the present disclosure.

[0036] The fifth aspect embodiment of the present disclosure proposes a computer program product, characterized in that it includes a computer program, and when the computer program is executed by a processor, it implements any method in the first aspect embodiment of the present disclosure.

[0037] The sixth aspect embodiment of the present disclosure proposes a chip, comprising at least one processor and a communication interface; the communication interface is used to receive signals input into the chip or signals output from the chip, the processor communicates with the communication interface and implements any one of the methods in the first aspect embodiment of the present disclosure through logic circuits or execution code instructions.

[0038] In summary, according to the three-dimensional target recognition method proposed in the present invention, based on the multi-view image of the target object, a two-dimensional backbone network is used to extract the multi-view and multi-scale image features of the target object, and comprehensive and accurate features are obtained, providing feature data for three-dimensional target recognition; sparse convolution is used to generate mutually correlated three-dimensional semantic Gaussians, and the three-dimensional semantic Gaussians include three-dimensional Gaussians of target object features, which efficiently represent the three-dimensional structural information of the three-dimensional target. At the same time, the mutual correlation between Gaussians is used to realize the encoding of spatial relationships and contextual information, and the understanding and expression capabilities of complex three-dimensional structures are improved; based on image features, the three-dimensional semantic Gaussians are updated through a deformable attention mechanism, and the physical properties of the three-dimensional semantic Gaussians are adjusted, which enhances the ability to handle complex and changing scenes and improves the accuracy and robustness of feature representation; based on the three-dimensional semantic Gaussians, the target object is identified through a Gaussian voxel sputtering algorithm, which realizes efficient and precise recognition of the target object and significantly improves the performance of three-dimensional object recognition.

[0039] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0041] Figure 1 A flowchart of a three-dimensional target recognition method according to an embodiment of the present disclosure;

[0042] Figure 2 A flowchart of generating mutually correlated three-dimensional semantic Gaussians through sparse convolution according to an embodiment of the present disclosure;

[0043] Figure 3 A flowchart of updating a three-dimensional semantic Gaussian through a deformable attention mechanism based on image features according to an embodiment of the present disclosure;

[0044] Figure 4 A flowchart of adjusting the physical properties of a three-dimensional semantic Gaussian according to an embodiment of the present disclosure;

[0045] Figure 5 A flowchart of an embodiment of the present disclosure for identifying a target object by a Gaussian voxel sputtering algorithm based on three-dimensional semantic Gaussian;

[0046] Figure 6 A flowchart of initializing a three-dimensional semantic Gaussian according to an embodiment of the present disclosure;

[0047] Figure 7 A schematic diagram of a processing process of a three-dimensional target recognition method according to an embodiment of the present disclosure;

[0048] Figure 8 A schematic diagram comparing the technical effects achieved by the embodiments of the present disclosure with the prior art;

[0049] Fig. 9 A schematic diagram of the structure of a three-dimensional target recognition device according to an embodiment of the present disclosure;

[0050] Fig.10 is a block diagram of an electronic device for implementing the three-dimensional target recognition method disclosed in the present invention according to an exemplary embodiment;

[0051] Fig.11 It is a schematic diagram of the structure of the chip of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0052] Embodiments of the present disclosure are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout identify the same or similar originals or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present disclosure, and should not be construed as limiting the present disclosure.

[0053] First, a brief introduction to the relevant terms in this disclosure is given:

[0054] 3D Gaussian: In the present disclosure, it refers to a 3D Gaussian distribution or a 3D Gaussian function, the general form of which can be expressed as:

[0055]

[0056] Among them, x is a point in three-dimensional space, μ is the mean vector (center point), and ∑ is the covariance matrix, which controls the shape and direction of the function. In three-dimensional space, a three-dimensional Gaussian can also be extended to an ellipsoid that obeys a Gaussian distribution. Its specific shape is determined by the covariance matrix ∑. If ∑ is a diagonal matrix, the Gaussian distribution in each dimension is independent; if there are off-diagonal elements, it indicates the correlation between different dimensions.

[0057] Voxel: In this disclosure, it refers to a volume element in three-dimensional space, similar to a pixel in a two-dimensional image. A voxel is a cubic unit in a three-dimensional data set, and each voxel contains information and attribute values ​​of its position in three-dimensional space.

[0058] 3D Gaussian Splatting algorithm: In this disclosure, it refers to a method for creating a 3D scene representation using a set of 2D images, the core of which is to convert a point cloud into an ellipsoid (3D Gaussian) in 3D space. The geometric properties (position, size), color, and opacity of these ellipsoids are optimized to mix together to generate a complete model that can be rendered from any angle. It estimates a sparse point cloud from a set of multi-view images through motion structure recovery technology, and then constructs a Gaussian ellipsoid probability prediction model for each point in the point cloud, and obtains the parameters of each ellipsoid through neural network learning, ultimately supporting multi-angle volume rendering and rasterization.

[0059] With the advancement of sensor technology and artificial intelligence technology, the recognition of objects has evolved from two-dimensional images to the recognition of three-dimensional data such as three-dimensional point clouds. In the task of three-dimensional object recognition, many three-dimensional object recognition algorithms have evolved, such as PointNet, PointCNN, 3D U-Net, NeRF and other algorithms, but usually due to the large amount of target object data and scattered data points, the performance of target recognition is often difficult to improve.

[0060] Especially in the autonomous driving scenario, whether to use LiDAR for 3D perception has always been a core issue of debate among autonomous driving companies due to the strong openness, high real-time performance, large number of targets and many types of targets in this scenario. Although vision-centric systems are more economical, related technologies cannot capture obstacles of arbitrary shapes, which hinders the safety and robustness of driving. The emergence of 3D object recognition methods solves this problem, which can predict the occupancy state of each voxel in the surrounding 3D space, thus providing a basis for various emerging tasks such as end-to-end autonomous driving, 4D occupancy prediction and self-supervised 3D scene understanding.

[0061] Despite its promising applications, the high-dimensional and dense output space of 3D occupancy prediction poses a great challenge to efficiently and effectively represent 3D scenes. Voxel-based methods assign a feature vector to each voxel to obtain a dense representation to describe the fine-grained structure of the 3D scene. Considering the sparsity of 3D space, these methods adopt coarse-to-fine upsampling or voxel filtering to improve efficiency.

[0062] In the related art, the voxel-based and bird's eye view (BEV)-based methods are essentially grid-based methods, so they cannot dynamically allocate computing and storage resources according to the sparsity and diversity of the scene, and may have problems of redundant grids and over-compression. That is, the related art has the problems of low recognition performance and high algorithm complexity in 3D object recognition.

[0063] The method proposed in the present disclosure is applied to three-dimensional target recognition tasks, and its application has a wide range of scenarios, and can be widely used in scenarios such as autonomous driving, robot navigation, three-dimensional reconstruction, intelligent monitoring, industrial automation, and virtual display. In the autonomous driving scenario, rapid and accurate identification of surrounding objects is crucial for safe driving; in the robot navigation scenario, this method can improve the robot's ability to perceive the environment, thereby achieving more effective path planning and obstacle avoidance. The intelligent monitoring system can also improve the detection accuracy of targets in the scene through this method, thereby improving overall safety. In the industrial automation scenario, this method can quickly identify and locate objects on the production line, improving production efficiency and the accuracy of quality control. In the field of virtual reality, through accurate three-dimensional target recognition technology, a more realistic virtual environment can be created to enhance the user's sense of immersion. The application scenario is not limited in the embodiments of the present disclosure.

[0064] The three-dimensional target recognition method provided by the present disclosure is described in detail below with reference to the accompanying drawings.

[0065] Figure 1 FIG. 1 is a flow chart of a three-dimensional target recognition method according to an embodiment of the present disclosure. Figure 1 In the embodiment shown, the three-dimensional target recognition method includes:

[0066] Step 101: Based on the multi-view images of the target object, a two-dimensional backbone network is used to extract multi-view and multi-scale image features of the target object.

[0067] In this embodiment, the target object refers to the object to be identified, including various objects or human bodies. Multi-view images refer to images taken at different positions of the target object. The two-dimensional backbone network refers to an image feature extraction network, which optionally includes a residual network (Residual Network, ResNet) and neural networks such as VGGNet, InceptionNet, and MobileNet. Based on the multi-view images taken from multiple different viewpoints of the target object, the image feature extraction network is used to extract multi-view and multi-scale image features of the target object.

[0068] Step 102: Generate mutually correlated three-dimensional semantic Gaussians through sparse convolution, where the three-dimensional semantic Gaussians include three-dimensional Gaussians of target object features.

[0069] In this embodiment, sparse convolution refers to a method of operation that introduces sparsity constraints into standard convolution operations to optimize the performance and resource usage of the model. A three-dimensional semantic Gaussian refers to a three-dimensional Gaussian used to describe the characteristics of a target object. Sparse convolution is performed on several initialized three-dimensional Gaussians to generate related three-dimensional semantic Gaussians.

[0070] Step 103: Based on the image features, the three-dimensional semantic Gaussian is updated through a deformable attention mechanism, and the physical properties of the three-dimensional semantic Gaussian are adjusted.

[0071] In this embodiment, the deformable attention mechanism refers to a technology used in deep learning to enhance the model's ability to understand image features. It is mainly extended on the basis of the self-attention mechanism, and allows attention operations to be performed on irregular, dynamically determined sampling points to improve the model's adaptability to changes in object shape and structure. The physical properties of a three-dimensional semantic Gaussian refer to the characteristics of describing and processing the shape, size, position, and probability distribution of an object in space. Specifically, the physical properties include parameters such as center position (mean vector), shape and orientation (covariance matrix), color, opacity, and spherical harmonic coefficients.

[0072] According to the extracted image features of the target object, the initialized 3D semantic Gaussian is updated through a deformable attention mechanism, and the physical properties of the 3D semantic Gaussian are adjusted so that the 3D semantic Gaussian is gradually iterated to the actual appearance of the 3D target object.

[0073] Step 104 : identifying the target object by using a Gaussian voxel sputtering algorithm based on the three-dimensional semantic Gaussian.

[0074] In this embodiment, for the 3D semantic Gaussian, a Gaussian voxel sputtering algorithm is used to construct a Gaussian ellipsoid probability prediction model for each 3D semantic Gaussian, and the parameters of each ellipsoid are obtained through neural network learning, ultimately supporting multi-angle volume rendering and rasterization. The target object is identified by combining the labels and weights of all 3D semantic Gaussians.

[0075] In summary, according to the three-dimensional target recognition method proposed in the present invention, based on the multi-view image of the target object, a two-dimensional backbone network is used to extract the multi-view and multi-scale image features of the target object, and comprehensive and accurate features are obtained, providing feature data for three-dimensional target recognition; sparse convolution is used to generate mutually correlated three-dimensional semantic Gaussians, and the three-dimensional semantic Gaussians include three-dimensional Gaussians of target object features, which efficiently represent the three-dimensional structural information of the three-dimensional target. At the same time, the mutual correlation between Gaussians is used to realize the encoding of spatial relationships and contextual information, and the understanding and expression capabilities of complex three-dimensional structures are improved; based on image features, the three-dimensional semantic Gaussians are updated through a deformable attention mechanism, and the physical properties of the three-dimensional semantic Gaussians are adjusted, which enhances the ability to handle complex and changing scenes and improves the accuracy and robustness of feature representation; based on the three-dimensional semantic Gaussians, the target object is identified through a Gaussian voxel sputtering algorithm, which realizes efficient and precise recognition of the target object and significantly improves the performance of three-dimensional object recognition.

[0076] Figure 2The present invention is a flowchart of generating mutually correlated three-dimensional semantic Gaussians through sparse convolution according to an embodiment of the present invention. Figure 2 Yes Figure 1 Further explanation of step 102 is based on Figure 2 The embodiment shown comprises the following steps:

[0077] Step 201, using the center of the three-dimensional semantic Gaussian as a spatial point, to generate a three-dimensional semantic point cloud.

[0078] In this embodiment, the 3D semantic point cloud refers to the point cloud corresponding to the 3D semantic Gaussian, and the spatial coordinates of the point cloud correspond to the center position of the 3D semantic Gaussian. The center of the 3D semantic Gaussian is taken as the spatial point, and the spatial point is taken as the position of the 3D semantic point cloud.

[0079] Step 202: voxelize the three-dimensional semantic point cloud into three-dimensional semantic voxels.

[0080] In this embodiment, the 3D semantic voxel refers to a structured representation of a 3D semantic point cloud. The 3D semantic point cloud is voxelized into 3D semantic voxels.

[0081] In one implementation of this embodiment, the spatial range of the three-dimensional semantic point cloud and the required resolution of the three-dimensional semantic voxels are determined. The spatial range can be defined by the minimum and maximum coordinates in the point cloud, and the resolution determines the size of the voxel grid, that is, the actual size of the space represented by each voxel. According to the defined spatial range and resolution, an empty three-dimensional array (voxel grid) is created, each array element represents a voxel, and is usually set to 0 or False during initialization to indicate blank or unoccupied. For each point in the point cloud, calculate where it should fall in the voxel grid, which can be achieved by dividing the coordinates of the point by the resolution and rounding. At the calculated voxel position, the corresponding voxel value is updated to 1 or True, indicating that the voxel is occupied by the point, thereby obtaining a three-dimensional semantic voxel.

[0082] Step 203: Based on the three-dimensional semantic voxels, three-dimensional sparse convolution is used to process the three-dimensional semantic voxels into associated three-dimensional semantic voxels, where the associated three-dimensional semantic voxels correspond to three-dimensional semantic Gaussians.

[0083] In this embodiment, the associated three-dimensional semantic voxel refers to the three-dimensional semantic voxel obtained by convolution processing, and the voxel corresponds to the three-dimensional semantic Gaussian. The three-dimensional semantic voxel obtained according to the above steps is processed using three-dimensional sparse convolution to obtain the associated three-dimensional semantic voxel. Preferably, the operation step length of the three-dimensional sparse convolution is 1.

[0084] In this embodiment, sparse convolution is used to generate interrelated 3D semantic Gaussians, taking into account the sparsity of 3D semantic Gaussians when modeling 3D scenes, and making full use of the interaction between 3D Gaussians in 3D scenes. The encoding of spatial relationships and contextual information of 3D semantic Gaussians is achieved, which improves the algorithm's ability to understand and express complex 3D structures.

[0085] Figure 3 This is a flowchart of an embodiment of the present disclosure for updating a three-dimensional semantic Gaussian based on image features through a deformable attention mechanism. Figure 3 Yes Figure 1 Further explanation of step 103 is based on Figure 3 The embodiment shown comprises the following steps:

[0086] Step 301 , based on the camera parameters corresponding to the multi-view images, each three-dimensional semantic Gaussian is projected into a pixel coordinate system through a projection transformation to generate a two-dimensional semantic image.

[0087] In this embodiment, the camera parameters corresponding to the multi-view images include internal parameters and external parameters. Based on the internal parameters and external parameters, the three-dimensional coordinate points in the world coordinate system can be converted to the target projection plane of the pixel coordinate system through projection transformation to form a two-dimensional semantic image.

[0088] Step 302, based on each pixel point of the two-dimensional semantic image, use a deformable attention mechanism to generate the offset and attention weight of the target pixel point in the pixel coordinate system to obtain a semantic offset feature map, and the target pixel point is any pixel point in the two-dimensional semantic image.

[0089] In this embodiment, the target pixel refers to any pixel in the two-dimensional semantic image currently being processed. The semantic bias feature map refers to the target pixel generating a bias after being processed by the deformable attention mechanism. Each pixel of the two-dimensional semantic image generates a corresponding bias, and all the biases form a matrix. For each pixel of the two-dimensional semantic image, the deformable attention mechanism is used to generate the bias and attention weight of the target pixel in the pixel coordinate point to obtain a semantic bias feature map.

[0090] Step 303, upsampling the semantic bias feature map by bilinear interpolation to obtain the features of the target reference point corresponding to the target three-dimensional semantic Gaussian in the semantic bias feature, wherein the target three-dimensional semantic Gaussian corresponds to the target pixel point, and each three-dimensional semantic Gaussian includes the target three-dimensional semantic Gaussian.

[0091] In this embodiment, the target three-dimensional semantic Gaussian refers to the three-dimensional Gaussian after the three-dimensional semantic Gaussian is updated and adjusted, which corresponds to the target pixel point, and the target three-dimensional semantic Gaussian belongs to the three-dimensional semantic Gaussian. The target reference point refers to the pixel point corresponding to the target pixel point in the two-dimensional semantic image in the upsampled bias feature map. The semantic bias feature map is upsampled by a bilinear interpolation algorithm to obtain the feature of the target reference point corresponding to the target three-dimensional semantic Gaussian in the semantic bias feature.

[0092] Step 304 , weighted summing is performed on the features of the target reference point according to the attention weight to obtain the features of the target three-dimensional semantic Gaussian, so as to update the three-dimensional semantic Gaussian.

[0093] In this embodiment, according to the attention weight of the target pixel point obtained in the above steps and the features of the corresponding target reference point, a weighted sum is performed on the two to obtain the features of the target three-dimensional semantic Gaussian. The feature corresponds to the three-dimensional semantic Gaussian and can be used to update the parameters of the three-dimensional semantic Gaussian, thereby enhancing the ability to handle complex and changing scenes.

[0094] In one implementation of this embodiment, for each three-dimensional semantic Gaussian, its mean position is transformed to the pixel coordinate system through projection transformation, and then deformable attention is used to generate an offset for each pixel coordinate. Then, bilinear interpolation sampling is performed on the image features according to the offset pixel coordinates to obtain the features of the corresponding reference points. Finally, the features of each reference point are weighted and summed based on the weight of the deformable attention to obtain the features of the corresponding three-dimensional semantic Gaussian.

[0095] Figure 4 The present invention is a flowchart of adjusting the physical properties of a three-dimensional semantic Gaussian according to an embodiment of the present invention. Figure 4 Yes Figure 1 Step 103 or Figure 3 The specific description after step 304 is based on Figure 4 The embodiment shown comprises the following steps:

[0096] Step 401, adjusting the physical properties of the three-dimensional semantic Gaussian, including: based on the characteristics of the three-dimensional semantic Gaussian, predicting the residual value corresponding to the parameter value of the physical property through a multi-layer perceptron, and updating the parameter value using the sum of the residual value and the parameter value to optimize the three-dimensional semantic Gaussian, the physical properties including covariance, color, opacity and spherical harmonic coefficients.

[0097] In this embodiment, in the process of updating the three-dimensional semantic Gaussian, in order to obtain a more accurate representation of the three-dimensional semantic Gaussian of the target object, it is also necessary to adjust the physical properties of the three-dimensional semantic Gaussian. The specific adjustment method includes: based on the characteristics of the three-dimensional semantic Gaussian, predicting the residual value corresponding to the parameter value of the physical attribute through a multi-layer perceptron, and updating the original parameter value using the sum of the residual value and the parameter value, thereby optimizing the three-dimensional semantic Gaussian. The physical properties of the three-dimensional semantic Gaussian include parameters such as covariance, color, opacity, and spherical harmonic coefficients.

[0098] In one implementation of this embodiment, at the initial stage of the model, the attributes (mean, covariance and semantic category) of each 3D semantic Gaussian are randomly initialized, and then these attributes are iteratively optimized layer by layer. In each layer of the network, a lightweight multilayer perceptron (MLP) is used to predict the residual value of each attribute from the features of the 3D semantic Gaussian, and then the residual value is added to the original attribute value of the 3D semantic Gaussian to complete the optimization of the layer. For example, the following formula The mean m of the three-dimensional semantic Gaussian is expressed by MLP through the residual value After correction, the adjusted three-dimensional semantic Gaussian feature G is obtained new .

[0099]

[0100] Where Q is the feature of the three-dimensional semantic Gaussian, and s, r, and c are the physical properties of the three-dimensional semantic Gaussian.

[0101] In this example, by adjusting the physical properties of the three-dimensional semantic Gaussian, the accuracy and robustness of the feature representation are improved, which helps to improve the accuracy of target object recognition.

[0102] Figure 5 The present invention is a flowchart of an embodiment of the present invention for identifying a target object through a Gaussian voxel sputtering algorithm based on three-dimensional semantic Gaussian. Figure 5 Yes Figure 2 and Figure 1 The specific description of step 104 is based on Figure 5 The embodiment shown comprises the following steps:

[0103] Step 501: Acquire three-dimensional semantic voxels.

[0104] In this embodiment, three-dimensional semantic voxels are obtained after three-dimensional sparse convolution operation.

[0105] Step 502: Determine a distance threshold corresponding to the three-dimensional semantic Gaussian according to the covariance of the three-dimensional semantic voxels.

[0106] In this embodiment, the distance threshold corresponding to the three-dimensional semantic Gaussian can be determined according to the covariance of the three-dimensional semantic voxels. The covariance of the three-dimensional semantic voxels reflects the density between different three-dimensional semantic voxels. If the covariance is large, it means that the three-dimensional semantic voxels are far apart and are relatively scattered as a whole; conversely, it means that the three-dimensional semantic voxels are densely distributed. Due to the relationship between the covariance and the distance of the three-dimensional semantic Gaussian, the distance threshold of the corresponding three-dimensional semantic Gaussian can be set by the covariance of the three-dimensional semantic voxels. Different covariance values ​​of three-dimensional semantic voxels set different distance thresholds between three-dimensional semantic Gaussians.

[0107] Step 503: if the distance between the center position of the 3D semantic Gaussian and the 3D semantic voxel is less than or equal to the distance threshold, the category of the 3D semantic voxel is used as the category label of the 3D semantic Gaussian; otherwise, the 3D semantic Gaussian is discarded.

[0108] In this embodiment, if the distance between the center position of the 3D semantic Gaussian and the 3D semantic voxel is less than or equal to the distance threshold, it means that the 3D semantic Gaussian is close to the 3D semantic voxel, and the category label of the 3D semantic voxel can be used as the category label of the 3D semantic Gaussian. Otherwise, it means that the distance between the two is too far, and the 3D semantic Gaussian is discarded, which greatly reduces the number of 3D semantic Gaussians that need to be queried for each voxel and improves the overall processing efficiency of the 3D semantic Gaussian.

[0109] Step 504: Use the category label as the recognition result of the target object.

[0110] In this embodiment, after classifying the 3D semantic Gaussians at different distances from the 3D semantic voxels, 3D semantic Gaussians with the same category label are obtained, and then these 3D semantic Gaussians are regarded as a whole, and the category label of the whole is used as the recognition result of the target object, so as to efficiently recognize the 3D target object by sight.

[0111] In one implementation of this embodiment, for any point in the three-dimensional space, the contribution of each three-dimensional semantic Gaussian to the point is considered, that is, the value of each multivariate Gaussian distribution at the point is weighted and summed to obtain the semantic occupancy result of the point:

[0112]

[0113] Among them, p is the coordinate of the three-dimensional semantic Gaussian in the three-dimensional space, m i is the mean of the center point of the i-th three-dimensional semantic Gaussian, c i is the semantic category of the i-th three-dimensional semantic Gaussian, and ∑ is the covariance between the three-dimensional semantic Gaussians. i (p; m, s, r, c) represents the predicted probability of voxel occupancy of the three-dimensional semantic Gaussian p under the conditions of m, s, r, and c. iThe sum of the predicted occupancy probabilities of P three-dimensional semantic Gaussians (p; m, s, r, c) from i=1 to i=P can obtain the corresponding category c of the target object.

[0114] Figure 6 The present invention is a flowchart of initializing a three-dimensional semantic Gaussian according to an embodiment of the present invention. Figure 6 Yes Figure 1 The specific description before step 102 is based on Figure 8 The embodiment shown comprises the following steps:

[0115] Step 601, generating at least two three-dimensional Gaussians obeying multivariate Gaussian distribution in three-dimensional space.

[0116] In this embodiment, before processing the three-dimensional semantic Gaussian, at least two three-dimensional Gaussians obeying multivariate Gaussian distribution are first generated in the three-dimensional space, and the parameters of the Gaussian distribution are randomly initialized.

[0117] Step 602: Initialize the three-dimensional semantic Gaussian using the three-dimensional Gaussian.

[0118] In this embodiment, the generated three-dimensional Gaussian is initialized as the three-dimensional semantic Gaussian.

[0119] In one implementation of this embodiment, sparse three-dimensional semantic Gaussians are used to model the three-dimensional space, each Gaussian represents a flexible local area and corresponds to a multivariate Gaussian distribution. The properties of the multivariate Gaussian distribution include mean, covariance and its semantic category:

[0120]

[0121] Where p is the coordinate of the 3D semantic Gaussian in the 3D space, m is the mean of the center point of the 3D semantic Gaussian, c is the semantic category of the 3D semantic Gaussian, and ∑ is the covariance between the 3D semantic Gaussians. g(p; m, s, r, c) represents the predicted probability of voxel occupancy of the 3D semantic Gaussian p under the conditions of m, s, r, and c.

[0122] This modeling method makes full use of the universal approximation property of the Gaussian mixture model to ensure the precision of the modeling; at the same time, the sparsity of the three-dimensional Gaussian also ensures the efficiency of the modeling.

[0123] Figure 7 FIG. 1 is a schematic diagram of a processing process of a three-dimensional target recognition method according to an embodiment of the present disclosure. Figure 7As shown, a multi-view image of the target object (crossroad) is collected, the image is encoded, and then the image feature extraction is performed and mapped with the three-dimensional semantic Gaussian to update the three-dimensional semantic Gaussian. First, the three-dimensional semantic Gaussian is initialized with the three-dimensional Gaussian, and the initial three-dimensional semantic Gaussian is self-encoded, that is, voxelized and sparsely convolved. Then, through the cross-attention mechanism of the image, the multi-view image of the three-dimensional semantic Gaussian projection is associated with the three-dimensional semantic Gaussian by establishing a reference point and a deformable attention mechanism, and the three-dimensional semantic Gaussian is updated, and then the interaction between different three-dimensional semantic Gaussians is established by traversal decoding and residual optimization. After repeated self-encoding, cross-attention mechanism and optimization, the three-dimensional semantic Gaussian of the target object is obtained, and then the occupancy prediction of the three-dimensional target object is identified by the Gaussian voxel sputtering algorithm to identify the category of the target object.

[0124] Figure 8 Schematic diagram for comparing the technical effects achieved by the embodiments of the present disclosure with the prior art. Figure 8 As shown in the figure, the voxel method represents the vehicle as shown in the voxel representation of Voxel, the BEV method represents the vehicle as shown in the bird's-eye view of BEV, the TPV method represents the vehicle in multiple views as shown in TPV, and the 3D Gaussian method represents the vehicle as shown in Figure 3 D Gaussian(ours). Compared with the four methods, the 3D Gaussian method itself has a very high sparse representation capability for three-dimensional space and can efficiently represent three-dimensional target objects. Through the method of the embodiment of the present disclosure, on the three-dimensional target recognition benchmark of the Occ3D-nuScenes dataset, the system achieved an mIoU of more than 19%; on the three-dimensional target recognition benchmark of the SSCBench-KITTI-360 dataset, the system achieved an mIoU of more than 12%.

[0125] The disclosed embodiment proposes a three-dimensional target recognition method. Based on the multi-view image of the target object, a two-dimensional backbone network is used to extract the multi-view and multi-scale image features of the target object, thereby obtaining comprehensive and accurate features and providing feature data for three-dimensional target recognition. Sparse convolution is used to generate mutually correlated three-dimensional semantic Gaussians, which include three-dimensional Gaussians of target object features, and efficiently represent the three-dimensional structural information of the three-dimensional target. At the same time, the mutual correlation between Gaussians is used to achieve the encoding of spatial relationships and contextual information, thereby improving the understanding and expression capabilities of complex three-dimensional structures. Based on image features, the three-dimensional semantic Gaussians are updated through a deformable attention mechanism, and the physical properties of the three-dimensional semantic Gaussians are adjusted, thereby enhancing the ability to handle complex and changing scenes and improving the accuracy and robustness of feature representation. According to the three-dimensional semantic Gaussians, the target object is identified through a Gaussian voxel sputtering algorithm, thereby achieving efficient and precise recognition of the target object and significantly improving the performance of three-dimensional object recognition.

[0126] Corresponding to the methods provided in the above-mentioned embodiments, the present disclosure also provides a three-dimensional target recognition device. Since the device provided in the embodiment of the present disclosure corresponds to the methods provided in the above-mentioned embodiments, the implementation method of the method is also applicable to the device provided in this embodiment and will not be described in detail in this embodiment.

[0127] Fig. 9 FIG. 9 is a schematic diagram of the structure of a three-dimensional object recognition device 900 according to an embodiment of the present disclosure. Fig. 9 As shown, the three-dimensional target recognition device includes:

[0128] A feature extraction module 910 is used to extract multi-view and multi-scale image features of the target object based on the multi-view images of the target object using a two-dimensional backbone network;

[0129] An association generation module 920, configured to generate mutually associated three-dimensional semantic Gaussians through sparse convolution, wherein the three-dimensional semantic Gaussians include three-dimensional Gaussians of target object features;

[0130] A 3D semantic Gaussian updating module 930 is used to update the 3D semantic Gaussian through a deformable attention mechanism based on image features and adjust the physical properties of the 3D semantic Gaussian;

[0131] The target recognition module 940 is used to recognize the target object through a Gaussian voxel sputtering algorithm according to the three-dimensional semantic Gaussian.

[0132] In some embodiments, the association generation module 920 is used to:

[0133] The center of the 3D semantic Gaussian is used as the spatial point to generate a 3D semantic point cloud;

[0134] voxelize the 3D semantic point cloud into 3D semantic voxels;

[0135] Based on the three-dimensional semantic voxels, three-dimensional sparse convolution is used to process the three-dimensional semantic voxels, and the associated three-dimensional semantic voxels correspond to three-dimensional semantic Gaussians.

[0136] The 3D semantic Gaussian updating module 930 updates the 3D semantic Gaussian based on the image features through a deformable attention mechanism in the following manner:

[0137] Based on the camera parameters corresponding to the multi-view images, each 3D semantic Gaussian is projected into the pixel coordinate system through projection transformation to generate a 2D semantic image.

[0138] Based on each pixel of the two-dimensional semantic image, a deformable attention mechanism is used to generate the offset and attention weight of the target pixel in the pixel coordinate system to obtain a semantic bias feature map. The target pixel is any pixel in the two-dimensional semantic image.

[0139] Upsampling the semantic bias feature map by bilinear interpolation to obtain the features of the target reference point corresponding to the target three-dimensional semantic Gaussian in the semantic bias feature, wherein the target three-dimensional semantic Gaussian corresponds to the target pixel point, and each three-dimensional semantic Gaussian includes the target three-dimensional semantic Gaussian;

[0140] The features of the target reference points are weighted and summed according to the attention weights to obtain the features of the target 3D semantic Gaussian to update the 3D semantic Gaussian.

[0141] In some embodiments, after updating the 3D semantic Gaussian, the 3D semantic Gaussian updating module 930 is further used to:

[0142] Adjust the physical properties of the 3D semantic Gaussian, including:

[0143] Based on the features of three-dimensional semantic Gaussian, a multi-layer perceptron is used to predict the residual value corresponding to the parameter value of the physical attribute, and the sum of the residual value and the parameter value is used to update the parameter value to optimize the three-dimensional semantic Gaussian. The physical attributes include covariance, color, opacity and spherical harmonic coefficients.

[0144] In some embodiments, the target identification module 940 is used to:

[0145] Obtaining three-dimensional semantic voxels;

[0146] According to the covariance of the three-dimensional semantic voxels, a distance threshold corresponding to the three-dimensional semantic Gaussian is determined;

[0147] If the distance between the center position of the 3D semantic Gaussian and the 3D semantic voxel is less than or equal to the distance threshold, the category of the 3D semantic voxel is used as the category label of the 3D semantic Gaussian, otherwise, the 3D semantic Gaussian is discarded;

[0148] The category label is used as the recognition result of the target object.

[0149] In some embodiments, before generating the mutually correlated three-dimensional semantic Gaussians through sparse convolution, the association generating module 920 is further configured to:

[0150] Generate at least two three-dimensional Gaussians that obey multivariate Gaussian distribution in three-dimensional space;

[0151] Initialize the 3D semantic Gaussian using the 3D Gaussian.

[0152] In summary, through the three-dimensional target recognition device, based on the multi-view image of the target object, a two-dimensional backbone network is used to extract the multi-view and multi-scale image features of the target object; sparse convolution is used to generate interrelated three-dimensional semantic Gaussians, which include three-dimensional Gaussians of target object features; based on image features, the three-dimensional semantic Gaussians are updated through a deformable attention mechanism, and the physical properties of the three-dimensional semantic Gaussians are adjusted; according to the three-dimensional semantic Gaussians, the target object is recognized through a Gaussian voxel sputtering algorithm. This device solves the problem of low three-dimensional target recognition performance in related technologies and improves the performance of three-dimensional object recognition.

[0153] In the above embodiments provided by the present disclosure, the methods and devices provided by the embodiments of the present disclosure are introduced. In order to implement the functions in the methods provided by the above embodiments of the present disclosure, the electronic device may include a hardware structure and a software module, and implement the above functions in the form of a hardware structure, a software module, or a hardware structure plus a software module. One of the above functions may be executed in the form of a hardware structure, a software module, or a hardware structure plus a software module.

[0154] Fig.10 is a block diagram of an electronic device 1000 for implementing the above three-dimensional object recognition method according to an exemplary embodiment.

[0155] For example, the electronic device 1000 may be a mobile phone, a computer, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0156] Reference Fig.10 , the electronic device 1000 may include one or more of the following components: a processing component 1002 , a memory 1004 , a power component 1006 , a multimedia component 1008 , an audio component 1010 , an input / output (I / O) interface 1012 , a sensor component 1014 , and a communication component 1016 .

[0157] The processing component 1002 generally controls the overall operation of the electronic device 1000, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 1002 may include one or more processors 1020 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 1002 may include one or more modules to facilitate the interaction between the processing component 1002 and other components. For example, the processing component 1002 may include a multimedia module to facilitate the interaction between the multimedia component 1008 and the processing component 1002.

[0158] The memory 1004 is configured to store various types of data to support operations on the electronic device 600. Examples of such data include instructions for any application or method operating on the electronic device 1000, contact data, phone book data, messages, pictures, videos, etc. The memory 1004 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0159] The power supply component 1006 provides power to various components of the electronic device 1000. The power supply component 1006 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 1000.

[0160] The multimedia component 1008 includes a screen that provides an output interface between the electronic device 1000 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 1008 includes a front camera and / or a rear camera. When the electronic device 1000 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0161] The audio component 1010 is configured to output and / or input audio signals. For example, the audio component 1010 includes a microphone (MIC), and when the electronic device 1000 is in an operation mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 1004 or sent via the communication component 1016. In some embodiments, the audio component 1010 also includes a speaker for outputting audio signals.

[0162] I / O interface 1012 provides an interface between processing component 1002 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.

[0163] The sensor assembly 1014 includes one or more sensors for providing various aspects of status assessment for the electronic device 1000. For example, the sensor assembly 1014 can detect the open / closed state of the electronic device 1000, the relative positioning of components, such as the display and keypad of the electronic device 1000, and the sensor assembly 1014 can also detect the position change of the electronic device 1000 or a component of the electronic device 1000, the presence or absence of user contact with the electronic device 1000, the orientation or acceleration / deceleration of the electronic device 1000, and the temperature change of the electronic device 1000. The sensor assembly 1014 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 1014 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 1014 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0164] The communication component 1016 is configured to facilitate wired or wireless communication between the electronic device 1000 and other devices. The electronic device 1000 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, 4G LTE, 5G NR (NewRadio) or a combination thereof. In an exemplary embodiment, the communication component 1016 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 1016 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0165] In an exemplary embodiment, the electronic device 1000 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0166] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 1004 including instructions, and the instructions can be executed by the processor 1020 of the electronic device 1000 to complete the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0167] The embodiments of the present disclosure further provide a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the three-dimensional object recognition method described in the above embodiments of the present disclosure.

[0168] An embodiment of the present disclosure further provides a computer program product, including a computer program, which executes the three-dimensional object recognition method described in the above embodiment of the present disclosure when a processor is used to execute the computer program.

[0169] Fig.11 is a schematic structural diagram of a chip 1100 for implementing the above three-dimensional object recognition method according to an exemplary embodiment.

[0170] Reference Fig.11 The chip 1100 includes at least one communication interface 1101 and a processor 1102; the communication interface 1101 is used to receive a signal input to the chip 1100 or a signal output from the chip 1100, and the processor 1102 communicates with the communication interface 1101 and implements the three-dimensional target recognition method described in the above embodiment through a logic circuit or executing code instructions.

[0171] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0172] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiments or examples are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0173] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present disclosure includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present disclosure belong.

[0174] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processing module, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (control method), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or otherwise processing in a suitable manner if necessary, and then stored in a computer memory.

[0175] It should be understood that the various parts of the embodiments of the present disclosure can be implemented by hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0176] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0177] In addition, each functional unit in each embodiment of the present disclosure may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium. The above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0178] Although the embodiments of the present disclosure have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present disclosure. A person skilled in the art may make changes, modifications, substitutions and variations to the above embodiments within the scope of the present disclosure.

Claims

1. A three-dimensional target recognition method, characterized in that: The method comprises: Based on the multi-view images of the target object, a two-dimensional backbone network is used to extract multi-view and multi-scale image features of the target object; Generate mutually correlated three-dimensional semantic Gaussians through sparse convolution, wherein the three-dimensional semantic Gaussians include the three-dimensional Gaussians of the target object features; Based on the image features, updating the three-dimensional semantic Gaussian through a deformable attention mechanism, and adjusting the physical properties of the three-dimensional semantic Gaussian; According to the three-dimensional semantic Gaussian, the target object is identified by a Gaussian voxel sputtering algorithm.

2. The method according to claim 1, characterized in that The method of generating mutually correlated three-dimensional semantic Gaussians by sparse convolution includes: Taking the center of the three-dimensional semantic Gaussian as a spatial point, generating a three-dimensional semantic point cloud; voxelize the three-dimensional semantic point cloud into three-dimensional semantic voxels; Based on the three-dimensional semantic voxels, three-dimensional sparse convolution is used to process the three-dimensional semantic voxels into associated three-dimensional semantic voxels, where the associated three-dimensional semantic voxels correspond to the three-dimensional semantic Gaussian.

3. The method according to claim 1, characterized in that The updating of the three-dimensional semantic Gaussian by a deformable attention mechanism based on the image features includes: Based on the camera parameters corresponding to the multi-view images, each of the three-dimensional semantic Gaussians is projected into a pixel coordinate system through a projection transformation to generate a two-dimensional semantic image; Based on each pixel point of the two-dimensional semantic image, using the deformable attention mechanism to generate an offset and an attention weight of a target pixel point in the pixel coordinate system to obtain a semantic offset feature map, wherein the target pixel point is any pixel point in the two-dimensional semantic image; Upsampling the semantic bias feature map by bilinear interpolation to obtain a feature of a target reference point corresponding to a target three-dimensional semantic Gaussian in the semantic bias feature, wherein the target three-dimensional semantic Gaussian corresponds to the target pixel point, and each of the three-dimensional semantic Gaussians includes the target three-dimensional semantic Gaussian; The features of the target reference point are weightedly summed according to the attention weight to obtain the features of the target three-dimensional semantic Gaussian, so as to update the three-dimensional semantic Gaussian.

4. The method according to claim 1 or 3, characterized in that: After the updating of the three-dimensional semantic Gaussian, the method further includes: Adjusting the physical properties of the three-dimensional semantic Gaussian includes: Based on the features of the three-dimensional semantic Gaussian, a multilayer perceptron is used to predict the residual value corresponding to the parameter value of the physical attribute, and the parameter value is updated using the sum of the residual value and the parameter value to optimize the three-dimensional semantic Gaussian, wherein the physical attributes include covariance, color, opacity, and spherical harmonic coefficients.

5. The method according to claim 2, characterized in that: The identifying the target object by using a Gaussian voxel sputtering algorithm according to the three-dimensional semantic Gaussian includes: Acquire the three-dimensional semantic voxel; Determining a distance threshold corresponding to the three-dimensional semantic Gaussian according to the covariance of the three-dimensional semantic voxel; If the distance between the center position of the 3D semantic Gaussian and the 3D semantic voxel is less than or equal to the distance threshold, the category of the 3D semantic voxel is used as the category label of the 3D semantic Gaussian; otherwise, the 3D semantic Gaussian is discarded; The category label is used as the recognition result of the target object.

6. The method according to claim 1, characterized in that Before generating the mutually correlated three-dimensional semantic Gaussians by sparse convolution, the method further includes: Generate at least two three-dimensional Gaussians that obey multivariate Gaussian distribution in three-dimensional space; The three-dimensional semantic Gaussian is initialized using the three-dimensional Gaussian.

7. A three-dimensional target recognition device, characterized in that: The device comprises: A feature extraction module, used to extract multi-view and multi-scale image features of the target object based on the multi-view image of the target object using a two-dimensional backbone network; An association generation module, used for generating mutually associated three-dimensional semantic Gaussians through sparse convolution, wherein the three-dimensional semantic Gaussians include the three-dimensional Gaussians of the target object features; A three-dimensional semantic Gaussian update module, used to update the three-dimensional semantic Gaussian through a deformable attention mechanism based on the image features and adjust the physical properties of the three-dimensional semantic Gaussian; The target recognition module is used to recognize the target object through a Gaussian voxel sputtering algorithm according to the three-dimensional semantic Gaussian.

8. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-6.

10. A computer program product, characterized in that The invention comprises a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 6.

11. A chip, characterized in that: It includes at least one processor and a communication interface; the communication interface is used to receive a signal input to the chip or a signal output from the chip, and the processor communicates with the communication interface and implements the method according to any one of claims 1-6 through a logic circuit or executing code instructions.

Citation Information

Cited By

  • Generalized efficient Gaussian representation learning method and device, equipment and storage medium

    CN119741423A

  • A generalizable efficient Gaussian representation learning method, device, equipment and storage medium

    CN119741423B

  • Multi-modal data fusion method and device based on dynamic Gaussian modeling and vehicle

    CN120259827A

  • Multimodal data fusion method, device and vehicle based on dynamic Gaussian modeling

    CN120259827B

  • Scene semantic occupancy prediction method based on semantic-distance adaptive Gaussian

    CN120673417A