Point cloud generation method and device based on multi-modal information, equipment and storage medium

Abstract

The application relates to the field of multi-modal technology, and discloses a point cloud generation method and device based on multi-modal information, equipment and a storage medium. The method comprises the following steps: generating a first loss of a point cloud generator according to a first predicted point cloud set, a first real point cloud set and a first loss model; generating a second loss of the point cloud generator according to a second predicted point cloud set, a second real point cloud set and a second loss model; and generating a third loss of the point cloud generator according to a third predicted point cloud set, a third real point cloud set and a third loss model. The total loss is generated according to the first loss, the second loss, the third loss and a total loss function. When the total loss meets a preset condition, the complete point cloud of the object to be completed is generated based on a cross-fusion module, a seed generator and an updated point cloud generator. The application can take into account the overall contour, main structure and local details of the object to be completed, and is beneficial to improving the generation effect of the complete point cloud of the object to be completed.

Description

Technical Field

[0001] This application relates to the field of multimodal technology, and in particular to a method, apparatus, device and storage medium for generating point clouds based on multimodal information. Background Technology

[0002] Complete point clouds of 3D objects have significant application value in fields such as 3D reconstruction, virtual modeling, intelligent detection, autonomous driving environmental perception, and digital twins, and can provide high-precision geometric data support for target recognition, scene understanding, and model rendering.

[0003] However, existing point cloud generation methods often suffer from problems such as missing structures, incomplete contours, and local collapses. They have a weak ability to grasp the overall shape of objects, and the worse their ability to generate overall contours, the more difficult it is to obtain point cloud data that is topologically complete, uniformly distributed, and closely matches the real structure. Therefore, how to generate a complete point cloud of the object to be completed is a technical problem that urgently needs to be solved. Summary of the Invention

[0004] This application provides a point cloud generation method, apparatus, device, and storage medium based on multimodal information to solve the aforementioned technical problem of how to generate a complete point cloud of an object to be completed.

[0005] In a first aspect, embodiments of this application provide a point cloud generation method based on multimodal information, applied to electronic devices, the point cloud generation method comprising:

[0006] Training samples are obtained from the sample set, which includes multimodal information of the preset object and the real point cloud of the preset object;

[0007] Based on the multimodal information of the preset object, the skeleton features of the preset object are determined. The skeleton features of the preset object are input into the point cloud generator, and the predicted point cloud of the preset object is generated by the point cloud generator.

[0008] Based on a predefined processing method, the predicted point cloud and the real point cloud of the preset object are processed to obtain the first set of predicted point clouds, the second set of predicted point clouds, the third set of predicted point clouds, the first set of real point clouds, the second set of real point clouds, and the third set of real point clouds.

[0009] Based on the first predicted point cloud set, the first real point cloud set, and the first loss model, a first loss of the point cloud generator is generated. Based on the second predicted point cloud set, the second real point cloud set, and the second loss model, a second loss of the point cloud generator is generated. Based on the third predicted point cloud set, the third real point cloud set, and the third loss model, a third loss of the point cloud generator is generated.

[0010] Based on the first loss, second loss, third loss and total loss function, the total loss of the point cloud generator is generated. When the total loss meets the preset conditions, the complete point cloud of the object to be completed is generated based on the cross-fusion module, the seed generator and the updated point cloud generator.

[0011] In one possible implementation of the first aspect, determining the skeleton features of the preset object based on its multimodal information, inputting the skeleton features of the preset object into a point cloud generator, and generating a predicted point cloud of the preset object through the point cloud generator includes:

[0012] From the multimodal information of the preset object, obtain the preset defect cloud and the text description information of the preset object. Input the text description information of the preset object into the large language model. Generate the enhanced description information of the preset object through the large language model. Input the text description information of the preset object into the diffusion model. Generate the basic features of the preset object through the diffusion model. Input the enhanced description information of the preset object into the diffusion model. Generate the component features of the preset object through the diffusion model. Input the preset defect cloud corresponding to the preset object into the projection network. Generate the geometric projection features of the preset object through the projection network.

[0013] The basic features, component features, and geometric projection features of the preset object are stitched together to generate the stitched features of the preset object. The stitched features of the preset object are then input into an image encoder to generate the two-dimensional visual features of the preset object. The defect cloud of the preset object is then input into a three-dimensional encoder to generate the three-dimensional geometric features of the preset object.

[0014] The two-dimensional visual features and three-dimensional geometric features of the preset object are input into the cross-fusion module, which generates the fused features of the preset object. The fused features of the preset object are then input into the seed generator, which generates the skeleton features of the preset object. The skeleton features of the preset object are then input into the point cloud generator, which generates the predicted point cloud of the preset object.

[0015] In one possible implementation of the first aspect, the process of processing the predicted point cloud and the real point cloud of the preset object based on a predefined processing method to obtain a first set of predicted point clouds, a second set of predicted point clouds, a third set of predicted point clouds, a first set of real point clouds, a second set of real point clouds, and a third set of real point clouds includes:

[0016] The farthest point sampling algorithm is used to sample the predicted point cloud of the preset object. 512 points are uniformly sampled from the predicted point cloud to form the first predicted point cloud set, 1024 points are uniformly sampled from the predicted point cloud to form the second predicted point cloud set, and 2048 points are uniformly sampled from the predicted point cloud to form the third predicted point cloud set.

[0017] The farthest point sampling algorithm is used to sample the real point cloud of the preset object. 512 points are uniformly sampled from the real point cloud to form the first real point cloud set, 1024 points are uniformly sampled from the real point cloud to form the second real point cloud set, and 2048 points are uniformly sampled from the real point cloud to form the third real point cloud set.

[0018] In one possible implementation of the first aspect, generating the total loss of the point cloud generator based on the first loss, second loss, third loss, and total loss function, and generating the complete point cloud of the object to be completed based on the cross-fusion module, seed generator, and updated point cloud generator when the total loss meets a preset condition, includes:

[0019] Based on the first loss, second loss, third loss and total loss model, the total loss of the point cloud generator is generated. The model parameters of the point cloud generator are updated through backpropagation until the total loss of the point cloud generator is less than the preset loss. Only then is the update of the model parameters of the point cloud generator stopped and the updated point cloud generator is saved.

[0020] Obtain the multimodal information of a preset object, and from the multimodal information of the preset object, obtain the defect cloud of the object to be completed and the text description information of the object to be completed;

[0021] The text description information of the object to be completed is input into the large language model, which generates the enhanced description information of the object to be completed. The text description information of the object to be completed is input into the diffusion model, which generates the basic features of the object to be completed. The enhanced description information of the object to be completed is input into the diffusion model, which generates the component features of the object to be completed. The defect cloud of the object to be completed is input into the projection network, which generates the geometric projection features of the object to be completed.

[0022] The basic features, component features, and geometric projection features of the object to be completed are stitched together to generate the stitched features of the object to be completed. These stitched features are then input into an image encoder to generate 2D visual features of the object. The defect cloud of the object to be completed is then input into a 3D encoder to generate 3D geometric features. The 2D visual features and 3D geometric features of the object to be completed are then input into a cross-fusion module to generate fused features. These fused features are then input into a seed generator to generate skeleton features. The skeleton features are then input into an updated point cloud generator to generate the complete point cloud of the object.

[0023] In one possible implementation of the first aspect, after generating the total loss of the point cloud generator based on the first loss, second loss, third loss, and total loss function, and generating the complete point cloud of the object to be completed based on the cross-fusion module, seed generator, and updated point cloud generator when the total loss meets a preset condition, the point cloud generation method includes:

[0024] Create a display window to show the complete point cloud of the object to be completed.

[0025] In one possible implementation of the first aspect, the total loss function is defined as follows:

[0026] ;

[0027] The total loss of the point cloud generator indicates the weakness of its ability to complete the overall outline, main structure, and local details of the preset object. The smaller the total loss, the stronger its ability to complete the overall outline, main structure, and local details of the preset object.

[0028] The first predicted point cloud set represents the preset object, and the first predicted point cloud set is used to complete the overall outline of the preset object.

[0029] The second predicted point cloud set represents the preset object, and the second predicted point cloud set is used to complete the main structure of the preset object;

[0030] This represents the third predicted point cloud set of the preset object, which is used to complete the local details of the preset object.

[0031] Represents the first set of real point clouds of the predefined object;

[0032] This represents the second set of real point clouds representing the predefined object;

[0033] This represents the third set of real point clouds for the predefined object.

[0034] In one possible implementation of the first aspect, the first loss model is defined as follows:

[0035]

[0036] This represents the first loss value of the point cloud generator. The smaller the first loss value of the point cloud generator, the stronger its ability to complete the overall outline of the preset object. The larger the first loss value of the point cloud generator, the weaker its ability to complete the overall outline of the preset object.

[0037] This represents the first predicted point cloud set of the preset object. Represents the first set of real point clouds of the predefined object;

[0038] This represents the total number of points in the first predicted point cloud set of the preset object; This represents the total number of points in the first real point cloud set of the preset object;

[0039] Represents any point in the first predicted point cloud set of the preset object; Represents any point in the first real point cloud set of the preset object;

[0040] Represents the nearest Euclidean distance from any point in the first predicted point cloud set of the preset object to the first real point cloud set;

[0041] Represents the nearest Euclidean distance from any point in the first true point cloud set of the preset object to the first predicted point cloud set;

[0042] The second loss model is defined as follows:

[0043]

[0044] This represents the second loss value of the point cloud generator. The smaller the second loss value of the point cloud generator, the stronger its ability to complete the main structure of the preset object. The larger the second loss value of the point cloud generator, the weaker its ability to complete the main structure of the preset object.

[0045] This represents the second predicted point cloud set of the preset object. This represents the second set of real point clouds representing the predefined object;

[0046] This represents the total number of points in the second predicted point cloud set for the preset object. This represents the total number of points in the second real point cloud set of the preset object;

[0047] Represents any point in the second predicted point cloud set of the preset object; Represents any point in the second real point cloud set of the preset object;

[0048] The nearest Euclidean distance from any point in the second predicted point cloud set of the preset object to the second real point cloud set is represented.

[0049] The nearest Euclidean distance from any point in the second true point cloud set of the preset object to the second predicted point cloud set is represented.

[0050] The third loss model is defined as follows:

[0051]

[0052] This represents the third loss value of the point cloud generator. The smaller the third loss value of the point cloud generator, the stronger its ability to complete the local details of the preset object. The larger the third loss value of the point cloud generator, the weaker its ability to complete the local details of the preset object.

[0053] This represents the third predicted point cloud set of the predefined object. Represents the third set of real point clouds of the predefined object;

[0054] This represents the total number of points in the third predicted point cloud set of the preset object; This represents the total number of points in the third real point cloud set of the preset object;

[0055] Represents any point in the third predicted point cloud set of the preset object; Represents any point in the third set of real point clouds of the preset object;

[0056] Represents the nearest Euclidean distance from any point in the third predicted point cloud set of the preset object to the third real point cloud set;

[0057] This represents the nearest Euclidean distance from any point in the third set of real point clouds of the preset object to the third set of predicted point clouds.

[0058] Secondly, embodiments of this application provide a point cloud generation device based on multimodal information, applied to electronic devices, including:

[0059] The acquisition module is used to acquire training samples from the sample set. The training samples include multimodal information of the preset object and the real point cloud of the preset object.

[0060] The determination module is used to determine the skeleton features of the preset object based on the multimodal information of the preset object, input the skeleton features of the preset object into the point cloud generator, and generate the predicted point cloud of the preset object through the point cloud generator.

[0061] The processing module is used to process the predicted point cloud and the real point cloud of the preset object based on a predefined processing method to obtain a first set of predicted point clouds, a second set of predicted point clouds, a third set of predicted point clouds, a first set of real point clouds, a second set of real point clouds, and a third set of real point clouds.

[0062] The first generation module is used to generate a first loss of the point cloud generator based on a first predicted point cloud set, a first real point cloud set and a first loss model; generate a second loss of the point cloud generator based on a second predicted point cloud set, a second real point cloud set and a second loss model; and generate a third loss of the point cloud generator based on a third predicted point cloud set, a third real point cloud set and a third loss model.

[0063] The second generation module is used to generate the total loss of the point cloud generator based on the first loss, the second loss, the third loss and the total loss function. When the total loss meets the preset conditions, the module generates the complete point cloud of the object to be completed based on the cross-fusion module, the seed generator and the updated point cloud generator.

[0064] Thirdly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the point cloud generation method described in the first aspect above.

[0065] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the point cloud generation method described in the first aspect above.

[0066] Fifthly, embodiments of this application provide a computer program product that, when run on an electronic device, causes the electronic device to execute the point cloud generation method described in the first aspect.

[0067] The beneficial effects of the embodiments of this application are as follows:

[0068] Firstly, based on the first loss, second loss, third loss and total loss function, the total loss of the point cloud generator is generated. When the total loss meets the preset conditions, the complete point cloud of the object to be completed is generated based on the cross-fusion module, seed generator and updated point cloud generator. This can effectively shorten the generation time of the complete point cloud of the object to be completed and is conducive to improving the generation efficiency of the complete point cloud of the object to be completed.

[0069] Secondly, the smaller the total loss of the point cloud generator, the stronger its ability to complete the overall outline, main structure, and local details of the preset object. The model parameters of the point cloud generator are updated through backpropagation until the total loss of the point cloud generator is less than the preset loss. Only then is the update of the model parameters of the point cloud generator stopped and the updated point cloud generator saved. Since the updated point cloud generator can take into account the overall outline, main structure, and local details of the object to be completed during the point cloud generation process, generating a complete point cloud of the object to be completed based on the updated point cloud generator is beneficial to improving the generation effect of the complete point cloud of the object to be completed. Attached Figure Description

[0070] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0071] Figure 1 This is an application scenario diagram of the point cloud generation method provided in the embodiments of this application;

[0072] Figure 2 This is a flowchart illustrating the point cloud generation method provided in an embodiment of this application;

[0073] Figure 3 A flowchart illustrating the implementation of S203 provided in this application embodiment;

[0074] Figure 4 A schematic block diagram of a point cloud generation apparatus provided in the embodiments of this application;

[0075] Figure 5 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application;

[0076] Figure 6 An example diagram illustrating the point cloud generation method provided in this application embodiment. Detailed Implementation

[0077] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0078] The point cloud generation method provided in this application can be applied to electronic devices, including but not limited to cloud servers, mobile phones, tablets, wearable devices, vehicle-mounted devices, and laptops. This application does not impose any restrictions on the specific type of electronic device.

[0079] Please see Figure 1 , Figure 1 The application scenario diagram of the point cloud generation method provided in the embodiments of this application is described in detail below:

[0080] Electronic devices access a storage server to retrieve training samples, which include multimodal information of a preset object and the real point cloud of the preset object.

[0081] In this embodiment, the electronic device obtains training samples uniformly from the storage server, which can ensure the consistency, integrity and reading stability of the training data, avoid training interruption due to local data loss, format disorder or loading delay, and provide sufficient and standardized training data for the point cloud generator.

[0082] Please see Figure 2 , Figure 2 This is a flowchart illustrating the point cloud generation method provided in this application embodiment, which can be applied to electronic devices.

[0083] like Figure 2 As shown, the point cloud generation method provided in this application includes the following steps, detailed below:

[0084] S201, Obtain training samples from the sample set. The training samples include multimodal information of the preset object and the real point cloud of the preset object.

[0085] The step of determining the skeleton features of a preset object based on its multimodal information, inputting these skeleton features into a point cloud generator, and generating a predicted point cloud of the preset object through the point cloud generator includes:

[0086] From the multimodal information of the preset object, obtain the preset defect cloud and the text description information of the preset object. Input the text description information of the preset object into the large language model. Generate the enhanced description information of the preset object through the large language model. Input the text description information of the preset object into the diffusion model. Generate the basic features of the preset object through the diffusion model. Input the enhanced description information of the preset object into the diffusion model. Generate the component features of the preset object through the diffusion model. Input the preset defect cloud corresponding to the preset object into the projection network. Generate the geometric projection features of the preset object through the projection network.

[0087] The basic features, component features, and geometric projection features of the preset object are stitched together to generate the stitched features of the preset object. The stitched features of the preset object are then input into an image encoder to generate the two-dimensional visual features of the preset object. The defect cloud of the preset object is then input into a three-dimensional encoder to generate the three-dimensional geometric features of the preset object.

[0088] The two-dimensional visual features and three-dimensional geometric features of the preset object are input into the cross-fusion module, which generates the fused features of the preset object. The fused features of the preset object are then input into the seed generator, which generates the skeleton features of the preset object. The skeleton features of the preset object are then input into the point cloud generator, which generates the predicted point cloud of the preset object.

[0089] For example, the two-dimensional visual features and three-dimensional geometric features of a preset object are input into the cross-fusion module, and the cross-fusion module generates fused features of the preset object, including:

[0090] The two-dimensional visual features and three-dimensional geometric features of the preset object are input into the cross-fusion module. The cross-fusion module uses multi-head cross attention to fuse the two-dimensional visual features and three-dimensional geometric features of the preset object to generate the fused features of the preset object.

[0091] S202, Based on the multimodal information of the preset object, determine the skeleton features of the preset object, input the skeleton features of the preset object into the point cloud generator, and generate the predicted point cloud of the preset object through the point cloud generator.

[0092] Among them, the point cloud generator is a module that automatically constructs 3D point cloud data of objects or scenes based on deep learning or generative models.

[0093] S203, based on a predefined processing method, process the predicted point cloud and the real point cloud of the preset object to obtain a first set of predicted point clouds, a second set of predicted point clouds, a third set of predicted point clouds, a first set of real point clouds, a second set of real point clouds, and a third set of real point clouds.

[0094] S204, Based on the first predicted point cloud set, the first real point cloud set and the first loss model, generate the first loss of the point cloud generator; based on the second predicted point cloud set, the second real point cloud set and the second loss model, generate the second loss of the point cloud generator; based on the third predicted point cloud set, the third real point cloud set and the third loss model, generate the third loss of the point cloud generator.

[0095] The first loss model is defined as follows:

[0096]

[0097] This represents the first loss value of the point cloud generator. The smaller the first loss value of the point cloud generator, the stronger its ability to complete the overall outline of the preset object. The larger the first loss value of the point cloud generator, the weaker its ability to complete the overall outline of the preset object.

[0098] This represents the first predicted point cloud set of the preset object. Represents the first set of real point clouds of the predefined object;

[0099] This represents the total number of points in the first predicted point cloud set of the preset object; This represents the total number of points in the first real point cloud set of the preset object;

[0100] Represents any point in the first predicted point cloud set of the preset object; Represents any point in the first real point cloud set of the preset object;

[0101] Represents the nearest Euclidean distance from any point in the first predicted point cloud set of the preset object to the first real point cloud set;

[0102] Represents the nearest Euclidean distance from any point in the first true point cloud set of the preset object to the first predicted point cloud set;

[0103] The second loss model is defined as follows:

[0104]

[0105] This represents the second loss value of the point cloud generator. The smaller the second loss value of the point cloud generator, the stronger its ability to complete the main structure of the preset object. The larger the second loss value of the point cloud generator, the weaker its ability to complete the main structure of the preset object.

[0106] This represents the second predicted point cloud set of the preset object. This represents the second set of real point clouds representing the predefined object;

[0107] This represents the total number of points in the second predicted point cloud set for the preset object. This represents the total number of points in the second real point cloud set of the preset object;

[0108] Represents any point in the second predicted point cloud set of the preset object; Represents any point in the second real point cloud set of the preset object;

[0109] The nearest Euclidean distance from any point in the second predicted point cloud set of the preset object to the second real point cloud set is represented.

[0110] The nearest Euclidean distance from any point in the second true point cloud set of the preset object to the second predicted point cloud set is represented.

[0111] The third loss model is defined as follows:

[0112]

[0113] This represents the third loss value of the point cloud generator. The smaller the third loss value of the point cloud generator, the stronger its ability to complete the local details of the preset object. The larger the third loss value of the point cloud generator, the weaker its ability to complete the local details of the preset object.

[0114] This represents the third predicted point cloud set of the predefined object. Represents the third set of real point clouds of the predefined object;

[0115] This represents the total number of points in the third predicted point cloud set of the preset object; This represents the total number of points in the third real point cloud set of the preset object;

[0116] Represents any point in the third predicted point cloud set of the preset object; Represents any point in the third set of real point clouds of the preset object;

[0117] Represents the nearest Euclidean distance from any point in the third predicted point cloud set of the preset object to the third real point cloud set;

[0118] This represents the nearest Euclidean distance from any point in the third set of real point clouds of the preset object to the third set of predicted point clouds.

[0119] S205. Based on the first loss, second loss, third loss and total loss function, generate the total loss of the point cloud generator. When the total loss meets the preset conditions, generate the complete point cloud of the object to be completed based on the cross-fusion module, seed generator and updated point cloud generator.

[0120] The step of generating the total loss of the point cloud generator based on the first loss, second loss, third loss, and total loss function, and generating the complete point cloud of the object to be completed based on the cross-fusion module, seed generator, and updated point cloud generator when the total loss meets preset conditions, includes:

[0121] Based on the first loss, second loss, third loss and total loss model, the total loss of the point cloud generator is generated. The model parameters of the point cloud generator are updated through backpropagation until the total loss of the point cloud generator is less than the preset loss. Only then is the update of the model parameters of the point cloud generator stopped and the updated point cloud generator is saved.

[0122] Obtain the multimodal information of a preset object, and from the multimodal information of the preset object, obtain the defect cloud of the object to be completed and the text description information of the object to be completed;

[0123] The text description information of the object to be completed is input into the large language model, which generates the enhanced description information of the object to be completed. The text description information of the object to be completed is input into the diffusion model, which generates the basic features of the object to be completed. The enhanced description information of the object to be completed is input into the diffusion model, which generates the component features of the object to be completed. The defect cloud of the object to be completed is input into the projection network, which generates the geometric projection features of the object to be completed.

[0124] The basic features, component features, and geometric projection features of the object to be completed are stitched together to generate the stitched features of the object to be completed. These stitched features are then input into an image encoder to generate 2D visual features of the object. The defect cloud of the object to be completed is then input into a 3D encoder to generate 3D geometric features. The 2D visual features and 3D geometric features of the object to be completed are then input into a cross-fusion module to generate fused features. These fused features are then input into a seed generator to generate skeleton features. The skeleton features are then input into an updated point cloud generator to generate the complete point cloud of the object.

[0125] The process involves inputting the skeleton features of the object to be completed into the updated point cloud generator, which then generates a complete point cloud of the object. This significantly reduces the cost and workload of point cloud acquisition, eliminating the need for secondary scanning or complex on-site acquisition of the object. It is particularly suitable for scenarios such as cultural relics and precision industrial parts, which are difficult to scan repeatedly and have high scanning costs. This approach reduces equipment and manpower investment while improving work efficiency.

[0126] For example, the two-dimensional visual features and three-dimensional geometric features of the object to be completed are input into the cross-fusion module, and the cross-fusion module generates the fused features of the object to be completed, including:

[0127] The two-dimensional visual features and three-dimensional geometric features of the object to be completed are input into the cross-fusion module. The cross-fusion module uses multi-head cross attention to fuse the two-dimensional visual features and three-dimensional geometric features of the object to be completed, generating the fused features of the object to be completed.

[0128] The point cloud generation method, after generating the total loss of the point cloud generator based on the first loss, second loss, third loss, and total loss function, and generating the complete point cloud of the object to be completed based on the cross-fusion module, seed generator, and updated point cloud generator when the total loss meets a preset condition, includes:

[0129] Create a display window to show the complete point cloud of the object to be completed.

[0130] The total loss function is defined as follows:

[0131] ;

[0132] The total loss of the point cloud generator indicates the weakness of its ability to complete the overall outline, main structure, and local details of the preset object. The smaller the total loss, the stronger its ability to complete the overall outline, main structure, and local details of the preset object.

[0133] The first predicted point cloud set represents the preset object, and the first predicted point cloud set is used to complete the overall outline of the preset object.

[0134] The second predicted point cloud set represents the preset object, and the second predicted point cloud set is used to complete the main structure of the preset object;

[0135] This represents the third predicted point cloud set of the preset object, which is used to complete the local details of the preset object.

[0136] Represents the first set of real point clouds of the predefined object;

[0137] This represents the second set of real point clouds representing the predefined object;

[0138] This represents the third set of real point clouds for the predefined object.

[0139] refer to Figure 6 , Figure 6 Example diagrams of the point cloud generation method provided in the embodiments of this application are described in detail below:

[0140] For example, the text description of the object to be completed is: a symmetrical chair with a square seat, four vertical legs, and a backrest made of vertical slats; the enhanced description of the object to be completed is:

[0141] 1. The seat is a horizontally placed square flat structure with slightly chamfered edges and a smooth surface.

[0142] 2. The chair legs are four square rod-shaped structures perpendicular to the ground, located under the four corners of the seat to provide stable support.

[0143] 3. The backrest is composed of multiple vertical slats arranged in parallel. The slats on both sides are slightly higher, and the overall shape is rectangular. It is installed above the rear edge of the seat.

[0144] Input the textual description information of the object to be completed into the diffusion model;

[0145] The basic features of the object to be completed are generated by the diffusion model. The enhanced description information of the object to be completed is input into the diffusion model, and the component features of the object to be completed are generated by the diffusion model.

[0146] The defect cloud of the object to be completed is input into the projection network, and the geometric projection features of the object to be completed are generated through the projection network.

[0147] The basic features, component features, and geometric projection features of the object to be completed are stitched together to generate the stitched features of the object to be completed. These stitched features are then input into an image encoder to generate the two-dimensional visual features of the object to be completed.

[0148] The defect cloud of the object to be completed is input into the 3D encoder, and the 3D encoder generates the 3D geometric features of the object to be completed.

[0149] The two-dimensional visual features and three-dimensional geometric features of the object to be completed are input into the cross-fusion module, which generates the fused features of the object to be completed.

[0150] The fusion features of the object to be completed are input into the seed generator, which generates the skeleton features of the object to be completed. The skeleton features of the object to be completed are then input into the updated point cloud generator, which generates the complete point cloud of the object to be completed.

[0151] The beneficial effects of the embodiments of this application are as follows:

[0152] Firstly, based on the first loss, second loss, third loss and total loss function, the total loss of the point cloud generator is generated. When the total loss meets the preset conditions, the complete point cloud of the object to be completed is generated based on the cross-fusion module, seed generator and updated point cloud generator. This can effectively shorten the generation time of the complete point cloud of the object to be completed and is conducive to improving the generation efficiency of the complete point cloud of the object to be completed.

[0153] Secondly, the smaller the total loss of the point cloud generator, the stronger its ability to complete the overall outline, main structure, and local details of the preset object. The model parameters of the point cloud generator are updated through backpropagation until the total loss of the point cloud generator is less than the preset loss. Only then is the update of the model parameters of the point cloud generator stopped and the updated point cloud generator saved. Since the updated point cloud generator can take into account the overall outline, main structure, and local details of the object to be completed during the point cloud generation process, generating a complete point cloud of the object to be completed based on the updated point cloud generator is beneficial to improving the generation effect of the complete point cloud of the object to be completed.

[0154] Please see Figure 3 , Figure 3 The implementation flowchart of S203 provided in the embodiments of this application is described in detail below:

[0155] S301, The farthest point sampling algorithm is used to sample the predicted point cloud of the preset object. 512 points are uniformly sampled from the predicted point cloud to form the first predicted point cloud set, 1024 points are uniformly sampled from the predicted point cloud to form the second predicted point cloud set, and 2048 points are uniformly sampled from the predicted point cloud to form the third predicted point cloud set.

[0156] Specifically, 512 points are uniformly sampled from the predicted point cloud, and these 512 points are aggregated together to form the first predicted point cloud set; 1024 points are uniformly sampled from the predicted point cloud, and these 1024 points are aggregated together to form the second predicted point cloud set; and 2048 points are uniformly sampled from the predicted point cloud, and these 2048 points are aggregated together to form the third predicted point cloud set.

[0157] S302, the farthest point sampling algorithm is used to sample the real point cloud of the preset object. 512 points are uniformly sampled from the real point cloud to form the first real point cloud set, 1024 points are uniformly sampled from the real point cloud to form the second real point cloud set, and 2048 points are uniformly sampled from the real point cloud to form the third real point cloud set.

[0158] Specifically, 512 points are uniformly sampled from the real point cloud, and these 512 points are aggregated together to form the first real point cloud set; 1024 points are uniformly sampled from the real point cloud, and these 1024 points are aggregated together to form the second real point cloud set; and 2048 points are uniformly sampled from the real point cloud, and these 2048 points are aggregated together to form the third real point cloud set.

[0159] The first set of real point clouds mainly selects points that can reflect the global 3D contour and overall geometric structure of the preset object. Its core function is to preserve global features and avoid losing the overall shape of the target. The second set of real point clouds corresponds to the sampling results at a medium scale. It selects points in the preset object that take into account both the global contour and local details, and serves as a bridge connecting the global and local to balance the integrity and precision of the sampling. The third set of real point clouds corresponds to the sampling results at a small scale. It focuses on the surface microstructure, local texture and other detailed features of the preset object to make up for the lack of detail depiction in large-scale sampling.

[0160] In this embodiment, multi-scale sampling of the real point cloud of a preset object is performed to generate a first set of real point clouds, a second set of real point clouds, and a third set of real point clouds. These sets allow the point cloud generator to fully learn the overall contour and local structural features of the preset object at different scales. This effectively improves the problem of poor overall contour generation capability in existing models due to the single scale of training data and insufficient global information. Through multi-scale data augmentation, the model can better grasp the spatial distribution pattern of the preset object from coarse-grained shape to fine-grained structure, avoiding contour incompleteness, local collapse, and structural distortion caused by missing scale information, thereby improving the training and generation quality of the point cloud generator.

[0161] For the point cloud generation method described in the above embodiments, please refer to [link / reference]. Figure 4 , Figure 4 This is a schematic block diagram of the point cloud generation apparatus provided in the embodiments of this application. Figure 4 The point cloud generation device 400 shown can be applied to, for example... Figure 1 The application scenario diagram shows electronic devices. The following section uses electronic devices as an example to illustrate this. Figure 4 The point cloud generation device 400 shown will be described in detail. The point cloud generation device 400 may include an acquisition module 401, a determination module 402, a processing module 403, a first generation module 404, and a second generation module 405.

[0162] The acquisition module 401 is used to acquire training samples from the sample set. The training samples include multimodal information of the preset object and the real point cloud of the preset object.

[0163] The determination module 402 is used to determine the skeleton features of the preset object based on the multimodal information of the preset object, input the skeleton features of the preset object into the point cloud generator, and generate the predicted point cloud of the preset object through the point cloud generator.

[0164] The processing module 403 is used to process the predicted point cloud and the real point cloud of the preset object based on a predefined processing method to obtain a first set of predicted point clouds, a second set of predicted point clouds, a third set of predicted point clouds, a first set of real point clouds, a second set of real point clouds, and a third set of real point clouds.

[0165] The first generation module 404 is used to generate a first loss of the point cloud generator based on a first predicted point cloud set, a first real point cloud set and a first loss model; generate a second loss of the point cloud generator based on a second predicted point cloud set, a second real point cloud set and a second loss model; and generate a third loss of the point cloud generator based on a third predicted point cloud set, a third real point cloud set and a third loss model.

[0166] The second generation module 405 is used to generate the total loss of the point cloud generator based on the first loss, the second loss, the third loss and the total loss function. When the total loss meets the preset conditions, it generates the complete point cloud of the object to be completed based on the cross-fusion module, the seed generator and the updated point cloud generator.

[0167] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0168] The beneficial effects of the embodiments of this application are as follows:

[0169] Firstly, based on the first loss, second loss, third loss and total loss function, the total loss of the point cloud generator is generated. When the total loss meets the preset conditions, the complete point cloud of the object to be completed is generated based on the cross-fusion module, seed generator and updated point cloud generator. This can effectively shorten the generation time of the complete point cloud of the object to be completed and is conducive to improving the generation efficiency of the complete point cloud of the object to be completed.

[0170] Secondly, the smaller the total loss of the point cloud generator, the stronger its ability to complete the overall outline, main structure, and local details of the preset object. The model parameters of the point cloud generator are updated through backpropagation until the total loss of the point cloud generator is less than the preset loss. Only then is the update of the model parameters of the point cloud generator stopped and the updated point cloud generator saved. Since the updated point cloud generator can take into account the overall outline, main structure, and local details of the object to be completed during the point cloud generation process, generating a complete point cloud of the object to be completed based on the updated point cloud generator is beneficial to improving the generation effect of the complete point cloud of the object to be completed.

[0171] Please see Figure 5 , Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0172] like Figure 5 As shown, Figure 5 The electronic device includes: at least one processor 20, a memory 21, and a computer program 22 stored in the memory 21 and executable on the at least one processor 20, wherein the processor 20 executes the computer program 22 to implement the steps in any of the above method embodiments.

[0173] The electronic device may include, but is not limited to, processor 20 and memory 21. Those skilled in the art will understand that... Figure 5 This is merely an example of an electronic device and does not constitute a limitation on electronic devices. It may include more or fewer components than shown in the illustration, or combinations of certain components, or different components. For example, it may also include input / output devices, network access devices, etc.

[0174] The processor 20 is used to run a computer program 22 stored in the memory 21, and performs the following steps when executing the computer program 22:

[0175] Training samples are obtained from the sample set, which includes multimodal information of the preset object and the real point cloud of the preset object;

[0176] Based on the multimodal information of the preset object, the skeleton features of the preset object are determined. The skeleton features of the preset object are input into the point cloud generator, and the predicted point cloud of the preset object is generated by the point cloud generator.

[0177] Based on a predefined processing method, the predicted point cloud and the real point cloud of the preset object are processed to obtain the first set of predicted point clouds, the second set of predicted point clouds, the third set of predicted point clouds, the first set of real point clouds, the second set of real point clouds, and the third set of real point clouds.

[0178] Based on the first predicted point cloud set, the first real point cloud set, and the first loss model, a first loss of the point cloud generator is generated. Based on the second predicted point cloud set, the second real point cloud set, and the second loss model, a second loss of the point cloud generator is generated. Based on the third predicted point cloud set, the third real point cloud set, and the third loss model, a third loss of the point cloud generator is generated.

[0179] Based on the first loss, second loss, third loss and total loss function, the total loss of the point cloud generator is generated. When the total loss meets the preset conditions, the complete point cloud of the object to be completed is generated based on the cross-fusion module, the seed generator and the updated point cloud generator.

[0180] The processor 20 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors, field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0181] In some embodiments, the memory 21 may be an internal storage unit of the electronic device, such as a hard disk or memory of the electronic device.

[0182] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0183] This application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the various method embodiments above.

[0184] The computer-readable storage medium may also be an external storage device of the point cloud generation device or electronic device, such as a plug-in hard drive, smart media card (SMC), secure digital (SD) card, flash card, or non-transitory computer-readable storage medium equipped on the point cloud generation device or electronic device.

[0185] Since the computer program stored in the computer-readable storage medium can execute any of the point cloud generation methods based on multimodal information provided in the embodiments of this application, the computer-readable storage medium can achieve the beneficial effects that any of the point cloud generation methods based on multimodal information provided in the embodiments of this application can achieve, as detailed in the preceding embodiments, and will not be repeated here.

[0186] This application provides a computer program product that, when run on an electronic device, causes the electronic device to execute the point cloud generation method described above.

[0187] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium.

[0188] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A point cloud generation method based on multimodal information, characterized in that, The point cloud generation method, applied to electronic devices, includes: Training samples are obtained from the sample set, which includes multimodal information of the preset object and the real point cloud of the preset object; From the multimodal information of a preset object, we obtain the preset defect cloud and text description information of the preset object. We input the text description information into a large language model to generate enhanced description information of the preset object. We then input the text description information into a diffusion model to generate the basic features of the preset object. Next, we input the enhanced description information into the diffusion model to generate component features of the preset object. Finally, we input the preset defect cloud into a projection network to generate geometric projection features of the preset object. This process is then used to refine the basic features, component features, and geometric projection features of the preset object. The projection features are stitched together to generate the stitched features of the preset object. The stitched features of the preset object are input into the image encoder, which generates the two-dimensional visual features of the preset object. The defect cloud of the preset object is input into the three-dimensional encoder, which generates the three-dimensional geometric features of the preset object. The two-dimensional visual features and the three-dimensional geometric features of the preset object are input into the cross-fusion module, which generates the fused features of the preset object. The fused features of the preset object are input into the seed generator, which generates the skeleton features of the preset object. The skeleton features of the preset object are input into the point cloud generator, which generates the predicted point cloud of the preset object. Based on a predefined processing method, the predicted point cloud and the real point cloud of the preset object are processed to obtain the first set of predicted point clouds, the second set of predicted point clouds, the third set of predicted point clouds, the first set of real point clouds, the second set of real point clouds, and the third set of real point clouds. Based on the first predicted point cloud set, the first real point cloud set, and the first loss model, a first loss of the point cloud generator is generated. Based on the second predicted point cloud set, the second real point cloud set, and the second loss model, a second loss of the point cloud generator is generated. Based on the third predicted point cloud set, the third real point cloud set, and the third loss model, a third loss of the point cloud generator is generated. Based on the first loss, second loss, third loss and total loss function, the total loss of the point cloud generator is generated. When the total loss meets the preset conditions, the complete point cloud of the object to be completed is generated based on the cross-fusion module, the seed generator and the updated point cloud generator.

2. The point cloud generation method according to claim 1, characterized in that, The process, based on a predefined processing method, processes the predicted point cloud and the actual point cloud of a preset object to obtain a first set of predicted point clouds, a second set of predicted point clouds, a third set of predicted point clouds, a first set of actual point clouds, a second set of actual point clouds, and a third set of actual point clouds, including: The farthest point sampling algorithm is used to sample the predicted point cloud of the preset object. 512 points are uniformly sampled from the predicted point cloud to form the first predicted point cloud set, 1024 points are uniformly sampled from the predicted point cloud to form the second predicted point cloud set, and 2048 points are uniformly sampled from the predicted point cloud to form the third predicted point cloud set. The farthest point sampling algorithm is used to sample the real point cloud of the preset object. 512 points are uniformly sampled from the real point cloud to form the first real point cloud set, 1024 points are uniformly sampled from the real point cloud to form the second real point cloud set, and 2048 points are uniformly sampled from the real point cloud to form the third real point cloud set.

3. The point cloud generation method according to claim 1, characterized in that, The step involves generating the total loss of the point cloud generator based on the first loss, second loss, third loss, and total loss function. When the total loss meets a preset condition, a complete point cloud of the object to be completed is generated based on the cross-fusion module, the seed generator, and the updated point cloud generator. This includes: Based on the first loss, second loss, third loss and total loss model, the total loss of the point cloud generator is generated. The model parameters of the point cloud generator are updated through backpropagation until the total loss of the point cloud generator is less than the preset loss. Only then is the update of the model parameters of the point cloud generator stopped and the updated point cloud generator is saved. Obtain the multimodal information of a preset object, and from the multimodal information of the preset object, obtain the defect cloud of the object to be completed and the text description information of the object to be completed; The text description information of the object to be completed is input into the large language model, which generates the enhanced description information of the object to be completed. The text description information of the object to be completed is input into the diffusion model, which generates the basic features of the object to be completed. The enhanced description information of the object to be completed is input into the diffusion model, which generates the component features of the object to be completed. The defect cloud of the object to be completed is input into the projection network, which generates the geometric projection features of the object to be completed. The basic features, component features, and geometric projection features of the object to be completed are stitched together to generate the stitched features of the object to be completed. These stitched features are then input into an image encoder to generate 2D visual features of the object. The defect cloud of the object to be completed is then input into a 3D encoder to generate 3D geometric features. The 2D visual features and 3D geometric features of the object to be completed are then input into a cross-fusion module to generate fused features. These fused features are then input into a seed generator to generate skeleton features of the object. The skeleton features are then input into an updated point cloud generator to generate the complete point cloud of the object.

4. The point cloud generation method according to claim 1, characterized in that, After generating the total loss of the point cloud generator based on the first loss, second loss, third loss, and total loss function, and generating the complete point cloud of the object to be completed based on the cross-fusion module, seed generator, and updated point cloud generator when the total loss meets a preset condition, the point cloud generation method includes: Create a display window to show the complete point cloud of the object to be completed.

5. The point cloud generation method according to claim 1, characterized in that, The total loss function is defined as follows: ； The total loss of the point cloud generator indicates the weakness of its ability to complete the overall outline, main structure, and local details of the preset object. The smaller the total loss, the stronger its ability to complete the overall outline, main structure, and local details of the preset object. The first predicted point cloud set represents the preset object, and the first predicted point cloud set is used to complete the overall outline of the preset object. The second predicted point cloud set represents the preset object, and the second predicted point cloud set is used to complete the main structure of the preset object; This represents the third predicted point cloud set of the preset object, which is used to complete the local details of the preset object. Represents the first set of real point clouds of the predefined object; This represents the second set of real point clouds representing the predefined object; This represents the third set of real point clouds for the predefined object.

6. The point cloud generation method according to claim 1, characterized in that, The first loss model is defined as follows: This represents the first loss value of the point cloud generator. The smaller the first loss value of the point cloud generator, the stronger its ability to complete the overall outline of the preset object. The larger the first loss value of the point cloud generator, the weaker its ability to complete the overall outline of the preset object. This represents the first predicted point cloud set of the preset object. Represents the first set of real point clouds of the predefined object; This represents the total number of points in the first predicted point cloud set of the preset object; This represents the total number of points in the first real point cloud set of the preset object; Represents any point in the first predicted point cloud set of the preset object; Represents any point in the first real point cloud set of the preset object; Represents the nearest Euclidean distance from any point in the first predicted point cloud set of the preset object to the first real point cloud set; Represents the nearest Euclidean distance from any point in the first true point cloud set of the preset object to the first predicted point cloud set; The second loss model is defined as follows: This represents the second loss value of the point cloud generator. The smaller the second loss value of the point cloud generator, the stronger its ability to complete the main structure of the preset object. The larger the second loss value of the point cloud generator, the weaker its ability to complete the main structure of the preset object. This represents the second predicted point cloud set of the preset object. This represents the second set of real point clouds representing the predefined object; This represents the total number of points in the second predicted point cloud set for the preset object. This represents the total number of points in the second real point cloud set of the preset object; Represents any point in the second predicted point cloud set of the preset object; Represents any point in the second real point cloud set of the preset object; The nearest Euclidean distance from any point in the second predicted point cloud set of the preset object to the second real point cloud set is represented. The nearest Euclidean distance from any point in the second true point cloud set of the preset object to the second predicted point cloud set is represented. The third loss model is defined as follows: This represents the third loss value of the point cloud generator. The smaller the third loss value of the point cloud generator, the stronger its ability to complete the local details of the preset object. The larger the third loss value of the point cloud generator, the weaker its ability to complete the local details of the preset object. This represents the third predicted point cloud set of the predefined object. Represents the third set of real point clouds of the predefined object; This represents the total number of points in the third predicted point cloud set of the preset object; This represents the total number of points in the third real point cloud set of the preset object; Represents any point in the third predicted point cloud set of the preset object; Represents any point in the third set of real point clouds of the preset object; Represents the nearest Euclidean distance from any point in the third predicted point cloud set of the preset object to the third real point cloud set; This represents the nearest Euclidean distance from any point in the third set of real point clouds of the preset object to the third set of predicted point clouds.

7. A point cloud generation device based on multimodal information, characterized in that, Applied to electronic devices, including: The acquisition module is used to acquire training samples from the sample set. The training samples include multimodal information of the preset object and the real point cloud of the preset object. The determination module is used to obtain the preset defect cloud and text description information of the preset object from the multimodal information of the preset object. The text description information of the preset object is input into a large language model to generate enhanced description information of the preset object. The text description information of the preset object is then input into a diffusion model to generate the basic features of the preset object. The enhanced description information of the preset object is then input into a diffusion model to generate the component features of the preset object. The preset defect cloud corresponding to the preset object is input into a projection network to generate the geometric projection features of the preset object. This process is then used to determine the basic features, component features, and other characteristics of the preset object. The geometric projection features of the object are stitched together to generate the stitched features of the preset object. The stitched features of the preset object are input into the image encoder, which generates the two-dimensional visual features of the preset object. The defect cloud of the preset object is input into the three-dimensional encoder, which generates the three-dimensional geometric features of the preset object. The two-dimensional visual features and the three-dimensional geometric features of the preset object are input into the cross-fusion module, which generates the fused features of the preset object. The fused features of the preset object are input into the seed generator, which generates the skeleton features of the preset object. The skeleton features of the preset object are input into the point cloud generator, which generates the predicted point cloud of the preset object. The processing module is used to process the predicted point cloud and the real point cloud of the preset object based on a predefined processing method to obtain a first set of predicted point clouds, a second set of predicted point clouds, a third set of predicted point clouds, a first set of real point clouds, a second set of real point clouds, and a third set of real point clouds. The first generation module is used to generate a first loss of the point cloud generator based on a first predicted point cloud set, a first real point cloud set and a first loss model; generate a second loss of the point cloud generator based on a second predicted point cloud set, a second real point cloud set and a second loss model; and generate a third loss of the point cloud generator based on a third predicted point cloud set, a third real point cloud set and a third loss model. The second generation module is used to generate the total loss of the point cloud generator based on the first loss, the second loss, the third loss and the total loss function. When the total loss meets the preset conditions, the module generates the complete point cloud of the object to be completed based on the cross-fusion module, the seed generator and the updated point cloud generator.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the point cloud generation method as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the point cloud generation method as described in any one of claims 1 to 7.

Images