Point cloud super-resolution method and device, electronic equipment and storage medium

Abstract

The application discloses a point cloud super-resolution method and device, electronic equipment and storage medium. The method comprises the following steps: constructing and training a point cloud initial super-resolution model to obtain a point cloud super-resolution model meeting an accuracy threshold; dividing a to-be-processed point cloud into fragmented point clouds, inputting the fragmented point clouds into the point cloud super-resolution model, extracting geometric information features and color information features of the fragmented point clouds through a generator, and reconstructing a super-resolution point cloud by the generator; judging whether the reconstructed super-resolution point cloud reaches a preset reliability threshold through a discriminator, and outputting the super-resolution point cloud reaching the preset reliability threshold. The method can reconstruct high-quality dense point cloud geometry and color information from sparse point cloud coordinates and colors, model the point cloud by graph convolution, solve the point cloud disorder problem, effectively utilize context information, integrate point cloud features, and obtain a high-resolution point cloud with accurate contour and clear details.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video image processing, and particularly relates to a point cloud super-resolution method and device, electronic equipment and storage medium. BACKGROUND

[0002] Super-resolution refers to a technology of improving original image resolution through hardware or software. With the rapid development of 3D acquisition technology, 3D sensors (such as LiDAR) are becoming more and more easily available, and the point cloud data obtained thereby is also more and more, and more and more researchers have gradually invested in the research of three-dimensional point cloud data. However, processing 3D point cloud instead of 2D pixel grid brings new challenges. Unlike the image space represented by a regular grid, point cloud has no spatial order and regular structure. The generated points should describe the basic geometry of the potential target object, which means that they should be roughly located on the surface of the target object. The generated points should be informative and should not be disorganized. The disorder and irregularity of point cloud data and the above difficulties limit the development of deep learning networks in the point cloud super-resolution task. The 3D convolution network analogous to image processing greatly limits the resolution of point cloud output, and the point-by-point convolution cannot effectively model the disorganized and structureless data of point cloud, lacking context information on point cloud. SUMMARY

[0003] The purpose of the present application is to provide a point cloud super-resolution method, device, electronic equipment and storage medium. In order to have a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This part is not a general review, nor is it intended to determine the key / important components or delineate the scope of protection of these embodiments. Its only purpose is to present some concepts in a simple form as a prelude to the detailed description that follows.

[0004] According to one aspect of the embodiments of the present application, a point cloud super-resolution method is provided, comprising:

[0005] constructing and training a point cloud initial super-resolution model to obtain a point cloud super-resolution model satisfying an accuracy threshold; the point cloud initial super-resolution model comprises a generator and a discriminator;

[0006] dividing a to-be-processed point cloud into fragmented point clouds, and inputting the fragmented point clouds into the point cloud super-resolution model;

[0007] extracting geometric information features and color information features of the fragmented point clouds by the generator, and reconstructing a super-resolution point cloud by the generator using the geometric information features and the color information features;

[0008] The discriminator judges whether the reconstructed super-resolution point cloud reaches a preset confidence threshold, and outputs the super-resolution point cloud reaching the preset confidence threshold.

[0009] In some embodiments of the present application, the generator comprises a parallel point cloud coordinate generation channel and a point cloud color generation channel; the point cloud coordinate generation channel is used to extract geometric information features of the fragmented point cloud; and the point cloud color generation channel is used to extract color information features of the fragmented point cloud.

[0010] In some embodiments of the present application, the training of the initial super-resolution model of the point cloud comprises:

[0011] The point cloud in the training set is input into the generator, and the point cloud coordinate generation channel and the point cloud color generation channel are subjected to two consecutive upsampling operations to obtain a predicted point cloud;

[0012] The prediction error of the predicted point cloud is calculated, the parameters in the generator are updated in reverse according to the prediction error, and the iteration is performed until the prediction error reaches a preset threshold, so as to obtain a generator after the first training is completed;

[0013] The point cloud in the training set and the corresponding predicted point cloud are input into the discriminator to judge the confidence of the predicted point cloud, and the entire initial super-resolution model of the point cloud is updated according to the confidence judgment result, and the iteration is performed until the confidence reaches a preset confidence threshold.

[0014] In some embodiments of the present application, the training of the initial super-resolution model of the point cloud comprises: using a composite loss as part of the super-resolution task training loss to train the initial super-resolution model of the point cloud; wherein the composite loss comprises an adversarial loss, a shape-aware loss, and a color loss based on geometric position.

[0015] In some embodiments of the present application, the shape-aware loss comprises an overall constraint and a detail constraint; the overall constraint uses EMD to calculate the distance between the data distribution of the predicted point cloud and the real point cloud for constraint; and the detail constraint uses the average distance between the predicted point cloud and the real point cloud and the density metric of the generated point cloud for constraint.

[0016] In some embodiments of the present application, the division of the to-be-processed point cloud into fragmented point clouds comprises:

[0017] The to-be-processed point cloud is evenly divided into small cubes with the same volume, and the fragmented sampling is performed in units of the small cubes; and when sampling, the points of adjacent cubes are collected at a preset overlap rate to obtain the fragmented point cloud.

[0018] In some embodiments of the present application, the extracting, by the generator, the geometric information features and the color information features of the fragmented point cloud comprises:

[0019] The geometric information features and the color information features of the fragmented point cloud are extracted by the generator using a three-dimensional space k-neighbor-based graph convolution operation.

[0020] According to another aspect of the embodiments of the present application, a point cloud super-resolution device is provided, comprising:

[0021] A training module is configured to construct and train a point cloud initial super-resolution model to obtain a point cloud super-resolution model satisfying an accuracy threshold; the point cloud initial super-resolution model comprises a generator and a discriminator;

[0022] A division module is configured to divide a to-be-processed point cloud into fragmented point clouds, and input the fragmented point clouds into the point cloud super-resolution model;

[0023] A reconstruction module is configured to extract, by the generator, geometric information features and color information features of the fragmented point cloud, and reconstruct a super-resolution point cloud by the generator using the geometric information features and the color information features;

[0024] A judgment output module is configured to judge, by the discriminator, whether the reconstructed super-resolution point cloud reaches a pre-set confidence threshold, and output the super-resolution point cloud reaching the pre-set confidence threshold.

[0025] According to another aspect of the embodiments of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the point cloud super-resolution method of any one of the above.

[0026] According to another aspect of the embodiments of the present application, a computer readable storage medium is provided, which stores a computer program executable by a processor to implement the point cloud super-resolution method of any one of the above.

[0027] One aspect of the embodiments of the present application provides a technical solution which can include the following beneficial effects:

[0028] The point cloud super-resolution method provided by the embodiments of the present application processes the fragmented point cloud by using a point cloud super-resolution model, extracts geometric information features and color information features of the fragmented point cloud by a generator, and reconstructs a super-resolution point cloud by the generator using the geometric information features and the color information features. Whether the reconstructed super-resolution point cloud reaches a preset reliability threshold is judged by a discriminator, and the super-resolution point cloud reaching the preset reliability threshold is output. The requirements of the upstream task of point cloud processing can be met, the problem of unordered point cloud is effectively solved, context information can be effectively utilized to integrate point cloud features, and a high-resolution point cloud with accurate contour and clear details can be obtained.

[0029] Other features and advantages of the present application will be illustrated in the following description, and some will become apparent from the description, or can be learned from the description, or can be determined without doubt from the description, or can be understood by implementing the embodiments of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0030] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments described in the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0031] Figure 1 A point cloud super-resolution method flowchart of some embodiments of the present application is shown;

[0032] Figure 2 A flowchart of training a point cloud initial super-resolution model in some embodiments of the present application is shown;

[0033] Figure 3 A point cloud super-resolution device structure block diagram of one embodiment of the present application is shown;

[0034] Figure 4 An electronic device structure block diagram of one embodiment of the present application is shown;

[0035] Figure 5 A schematic diagram of a computer readable storage medium of one embodiment of the present application is shown.

[0036] The implementation of the purpose of the present application, functional characteristics and advantages will be further described with reference to the drawings. DETAILED DESCRIPTION

[0037] In order to make the purposes, technical solutions and advantages of the present application clearer, further explanation will be given below in conjunction with the drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of the present application.

[0038] Those skilled in the art can understand that, unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art in the field of the present application. It should also be understood that terms such as those defined in a general dictionary should be understood to have meanings consistent with those in the context of the prior art, and should not be interpreted to have idealized or overly formal meanings unless specifically defined as such herein.

[0039] The original point cloud data is an unprocessed large complete point cloud, the input point cloud of the generator in the model is a sparse patch point cloud, and the output of the generator is a dense patch point cloud. The complete point cloud is obtained by aggregating all patch point clouds.

[0040] As shown in Figure 1 One embodiment of the present application provides a point cloud super-resolution method, which in some embodiments of the present embodiment includes steps S10 to S40:

[0041] S10, constructing and training a point cloud initial super-resolution model to obtain a point cloud super-resolution model satisfying a precision threshold. The point cloud initial super-resolution model includes a generator and a discriminator.

[0042] The generator includes a parallel point cloud coordinate generation channel and a point cloud color generation channel; the point cloud coordinate generation channel is used to extract geometric information features of the patch point cloud; and the point cloud color generation channel is used to extract color information features of the patch point cloud.

[0043] In some embodiments, the parallel channels of the generator, i.e. the point cloud coordinate generation channel and the point cloud color generation channel, both channels of different attributes take structure-aware graph convolution as the basic operation to constitute channel sub-modules of different layer structures. When extracting features, the two parallel channels fuse the features of different attributes, and then find a similar point set to form a graph structure with the fused high-dimensional features as a representation, and then perform the next step of feature extraction.

[0044] As shown in Figure 2 In some embodiments, the training of the point cloud initial super-resolution model includes:

[0045] S101. Input the point cloud from the training set into the generator, and perform two consecutive upsampling operations on the point cloud coordinate generation channel and the point cloud color generation channel to obtain the predicted point cloud.

[0046] S102. Calculate the prediction error of the predicted point cloud, update the parameters in the generator in reverse according to the prediction error, and iterate repeatedly until the prediction error reaches a preset threshold to obtain the generator after the first training is completed.

[0047] S103. Input the point cloud in the training set and the corresponding predicted point cloud into the discriminator to determine the confidence level of the predicted point cloud. Update the initial super-resolution model of the entire point cloud according to the confidence level determination result. Iterate until the confidence level reaches the preset confidence threshold.

[0048] In some implementations, the training of the initial super-resolution model of the point cloud includes: training the initial super-resolution model of the point cloud using a composite loss as part of the training loss for the super-resolution task; wherein the composite loss includes adversarial loss, shape-aware loss, and color loss based on geometric location.

[0049] In some implementations, the shape-aware loss includes global constraints and detail constraints; the global constraints are constrained by calculating the distance between the two data distributions, the predicted point cloud and the real point cloud, using EMD; the detail constraints are constrained by the average distance between point-to-point pairs between the predicted point cloud and the real point cloud and the density metric of the generated point cloud.

[0050] In some implementations, the initial super-resolution model for training point clouds is divided into two parts: training the generator and training the generator and discriminator together.

[0051] The first part trains the generator by taking a piece of point cloud as input (containing geometric coordinates and color) and performing two consecutive upsampling module operations on two parallel channels to obtain a high-resolution color point cloud with a 4x upsampling rate. The error between the generator's predicted point cloud and the real point cloud is calculated, and the parameters of the feature extraction module and the upsampling module in the generator are updated in reverse. This process is repeated until the generator training is complete.

[0052] The second part trains both the generator and the discriminator simultaneously. The generator takes a point cloud of the same size as the one in the first part as input, and the two parallel channels perform two upsampling operations to obtain a high-resolution point cloud. The discriminator takes the real point cloud and the generator's predicted point cloud as input, continuously samples the same point cloud and judges whether the generated point cloud is real or fake. At this time, the entire network is updated simultaneously. The generator is updated based on the error between the real point cloud and the predicted point cloud, and the discriminator is updated based on the correctness of the predicted point cloud. This process is repeated until the training of the entire model is completed.

[0053] S20, dividing the to-be-processed point cloud into a piece point cloud, and inputting the piece point cloud into the point cloud super-resolution model.

[0054] In some embodiments, the dividing the to-be-processed point cloud into a piece point cloud comprises:

[0055] The to-be-processed point cloud is uniformly divided into small cubes with the same volume, and piece sampling is performed in units of the small cubes, and when sampling, points of adjacent cubes are collected with a preset overlap rate to obtain the piece point cloud.

[0056] For example, parameters of the smallest cube space occupied by the complete point cloud are counted, the original point cloud is uniformly divided into small cubes with the same volume according to the parameters, piece sampling is performed in units of the divided small cubes, and when sampling, points of adjacent cubes are collected with a certain overlap rate to prevent the edge information of the small cubes from being lost.

[0057] In some embodiments, the input data is obtained by piece sampling the original data in a manner that the cubes have a certain overlap with each other, and the point cloud coordinates and color information are normalized correspondingly, facilitating the learning of the model.

[0058] S30, extracting geometric information features and color information features of the piece point cloud by the generator, and reconstructing a super-resolution point cloud by the generator using the geometric information features and the color information features.

[0059] In some embodiments, the extracting the geometric information features and the color information features of the piece point cloud by the generator comprises:

[0060] The geometric information features and the color information features of the piece point cloud are extracted by the generator using a graph convolution operation based on three-dimensional space k-nearest neighbors.

[0061] For example, the point cloud attribute information (coordinates and color) input by the model is first processed by a conventional graph convolution operation based on three-dimensional space k-nearest neighbors to preliminarily process the geometric and color attributes of the point cloud, and high-dimensional space features about the two attributes are obtained. This feature is used as the input of the structure-aware graph convolution.

[0062] In some embodiments, the geometric coordinates and the color information of the piece point cloud are used as input, the generator uses structure-aware graph convolution as a basic operation to extract features of the point cloud, and in addition, the generator adopts a parallel network framework and fuses multiple attribute information to jointly complete the super-resolution of the geometry and the color.

[0063] The method of structure-aware graph convolution used by the generator utilizes the similarity of high-dimensional space to find point sets with similar high-dimensional space structures, uses these point sets to construct local graphs for graph convolution, and adds residual between convolutions to extract more relevant and effective features and prevent feature loss. The structure-aware graph convolution is as defined in formula (1.1)

[0064]

[0065] wherein, is the center point of the i-th local graph of the l+1-th layer, is the j-th similar point sorted by similarity, is the set of all structure-similar points related to the center point, and σ is an activation function. W l and U l are learnable parameters of the center point and local graph of the l-th layer, b l is the bias term of the l-th layer.

[0066] The similarity measure of high-dimensional structure adopts cosine similarity. As shown in definition formula (1.2),

[0067]

[0068] wherein, x i represents the i-th dimensional feature value of a point feature in the point cloud, and y i represents the i-th dimensional feature value of another point in the point cloud.

[0069] The discriminator in the generative adversarial network determines the sampling times according to the number of input point clouds, performs continuous downsampling and judgment, and obtains the final true or false value of the input point cloud.

[0070] In some embodiments, according to the structure of the network itself used, the present application uses an adversarial loss as part of the super-resolution task training loss, which is defined as formula (1.3) and formula (1.4),

[0071]

[0072]

[0073] wherein, y represents the true value, represents the predicted value of the generator, represents the confidence value evaluated by the discriminator. The purpose of the generator is to deceive the discriminator by minimizing L gan (G) to generate a predicted value.

[0074] The shape-aware loss is used in the geometry loss in the training process. The shape-aware loss contains two parts of constraints, the overall and the details. The overall uses the EMD to calculate the distance between the data distribution of the predicted point cloud and the real point cloud to constrain; the details use the average distance of each point pair between the predicted point cloud and the real point cloud, the CD distance, and the density measurement of the generated point cloud to constrain. Through the dual constraints of the overall and the details, a high-quality dense point cloud with clear contours, complete details and uniform distribution can be obtained. The EMD distance and the CD distance are shown in formula (1.5) and formula (1.6),

[0075]

[0076]

[0077] wherein S1 and S2 are the real point cloud and the predicted point cloud respectively, is the mapping relationship from the real point cloud to the predicted point cloud obtained according to the specific point cloud.

[0078] The shape-aware loss is defined as formula (1.7),

[0079] L shape (S1,S2)=λ CD ·L CD (S1,S2)+λ EMD ·L EMD (S1,S2)+λ uni ·L uni , (1.7)

[0080] wherein L uni is the uniform distribution loss, λ CD , λ EMD and λ uni are the hyperparameters of the corresponding items. The values of these hyperparameters are set according to specific experiments.

[0081] In some embodiments, the color attribute of the point cloud is constrained in combination with the geometric attribute of the point cloud to achieve an accurate constraint effect. The constraint of the color of the point cloud uses the L2 distance to calculate the distance of the color rgb of the corresponding points to constrain the generation of the color of the point cloud according to the geometric correspondence of the point cloud. The color loss based on the geometric position is defined as formula (1.8),

[0082]

[0083] wherein c1 is the value of the predicted RGB, and c2 is the real RGB color value of the corresponding point cloud.

[0084] In some embodiments, the composite loss includes the adversarial loss, the shape-aware loss and the color loss based on the geometric position, and the composite relationship is shown in formula (1.9), LG The composition is shown in formula (1.10),

[0085] L total = L gan (G) + L gan (D) + λ shape L shape + λ color L color , (1.9)

[0086] L G = λ gan L gan (G) + λ shape L shape + λ color L color , (1.10)

[0087] wherein λ gan , λ shape and λ color are corresponding weight terms.

[0088] In some embodiments, the coordinates, colors and preliminary extracted geometric and color information features of the point cloud are taken as inputs. The basic features of different attributes are obtained by connecting the original data and the preliminary features, and the structure-aware graph convolution in the parallel channel is further learning and extraction of the basic features. The structure-aware graph convolution calculates the structure similar point set to obtain the structure similar local graph required for convolution according to the high-dimensional features of the point cloud attributes extracted in the previous step, and then performs convolution operation. By analogy, each time the structure-aware graph convolution is performed, the structure similar local graph is obtained according to the high-dimensional feature output of the upper layer, and the graph convolution operation is performed. The two attribute channels only perform feature fusion when calculating the structure similar local graph, and obtain the structure similar local graph complementary to the attributes, so as to remove the structure deviated points in the local graph between the two attributes, so as to obtain the most relevant structure similar graph and complete the graph convolution.

[0089] S40, determining whether the reconstructed super-resolution point cloud reaches the pre-set confidence threshold through the discriminator, and outputting the super-resolution point cloud reaching the pre-set confidence threshold.

[0090] In some embodiments, the discriminator takes the predicted patch point cloud as input, continuously downsamples the unified patch point cloud multiple times, judges the true and false of the point cloud under different conditions, uses multiple values, and finally determines the output of the confidence.

[0091] Experiments are performed on the method proposed in the present embodiment and various methods of the prior art, and the experiments are performed in a hardware environment of NVIDIA GeForce-GTX 2080 GPU and a software environment of Python.

[0092] The dataset used in this experiment is JPEG Pleno Database. JPEG Pleno Database is a public point cloud dataset. The dynamic voxelized point cloud sequences in this dataset are called 8i voxelized full bodies (8iVFB). There are four sequences in the dataset, namely long skirt, war booty, red and black soldier, and soldier. In each sequence, 42 RGB cameras are configured into 14 clusters to capture the entire body of a human body at a speed of 30 fps in 10s of time. A spatial resolution is provided for each sequence: a 1024*1024*1024 voxel cube, called depth 10. Each person has 300 frames of point clouds in different poses.

[0093] In this simulation experiment, the method is compared with a variety of super-resolution methods in the same dataset.

[0094] The following table is a statistical table of the comparison of various evaluation indexes of the method of this embodiment and seven other geometric super-resolution methods.

[0095] The CD in the following table is chamfer distance, EMD is earth mover's distance, and HD is Hausdorff Distance. JSD measures the similarity of two distributions, PRE→GT error calculates the average squared distance from each point in the prediction to the nearest point in the ground truth. GT→PRE error calculates the average squared distance from each point in the ground truth to the nearest point in the prediction. F-score is a comprehensive evaluation index calculated from precision and recall.

[0096] Table 1 Comparison of super-resolution distance index results

[0097] Method CD EMD HD PCL-Upsample 0.0175 0.0220 0.1296 Fc 0.1005 0.2128 1.3662 PCN-CD 0.0801 0.1808 0.5499 PCN-EMD 0.1579 0.1869 0.5572 Folding 0.1443 0.2220 0.9099 PU-Net 0.1742 0.1314 1.6710 AR-GCN 0.0208 0.0349 0.1527 The method of the invention 0.0057 0.0233 0.0368

[0098] Table 2 Comparison of super-resolution non-distance index results

[0099]

[0100] Since the methods suitable for geometric super-resolution are generally not suitable for color super-resolution and there are relatively few methods directly aimed at the color super-resolution task, the following experiments are designed according to the experimental requirements.

[0101] The following table is a statistical table of the comparison of various evaluation indexes of the method of this application and three other color super-resolution methods in the prior art.

[0102] Table 3

[0103]

[0104]

[0105] Table 4

[0106]

[0107] As can be seen from the result table, the evaluation indexes of the method of the embodiment are all better than those of other existing methods.

[0108] The point cloud super-resolution method provided in the embodiment of the application, the generator combines the multi-attribute information of the point cloud, uses the complementarity between the attributes, constructs a local graph similar in structure, thereby obtaining more effective high-order features of the point cloud, which is conducive to the dense reconstruction of the geometric coordinates and colors of the point cloud; the discriminator is in the form of multiple samplings of a single point cloud, which improves the accuracy of the judgment of the discriminator and is conducive to the overall training of the network model; after the model training is completed, the generator can reconstruct high-quality dense point cloud geometry and color information from sparse point cloud coordinates and colors, which can meet the requirements of the upstream tasks of point cloud processing; the graph convolution models the point cloud with a graph, effectively solves the unordered problem of the point cloud, and can effectively use context information to integrate the features of the point cloud, which is a relatively effective convolution operation method in point cloud processing; compared with the existing super-resolution method, the embodiment of the application obtains more relevant point sets to form a local graph by using structure-aware graph convolution, which is conducive to the acquisition of high-dimensional features of the point cloud; according to the complementarity between the attributes, more relevant graph information is used through information fusion to obtain local consistent and relevant high-dimensional features; a composite loss is used to constrain the training of the super-resolution network to obtain high-resolution point clouds with accurate outlines and clear details.

[0109] As shown in Figure 3 Another embodiment of the application provides a point cloud super-resolution device, which comprises:

[0110] A training module is configured to construct and train a point cloud initial super-resolution model to obtain a point cloud super-resolution model meeting a precision threshold; the point cloud initial super-resolution model comprises a generator and a discriminator;

[0111] A division module is configured to divide a point cloud to be processed into fragmented point clouds and input the fragmented point clouds into the point cloud super-resolution model;

[0112] A reconstruction module is configured to extract geometric information features and color information features of the fragmented point clouds by the generator and reconstruct a super-resolution point cloud by the generator using the geometric information features and the color information features;

[0113] A judgment output module is configured to judge whether the reconstructed super-resolution point cloud meets a preset reliability threshold by the discriminator and output the super-resolution point cloud meeting the preset reliability threshold.

[0114] The point cloud super-resolution device provided by the embodiments of the present application and the point cloud super-resolution method provided by the embodiments of the present application have the same inventive concept, and have the same beneficial effects as the method adopted, run or implemented by the point cloud super-resolution device.

[0115] Another embodiment of the present application provides an electronic device including a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor executes the program to implement the point cloud super-resolution method of any of the above embodiments.

[0116] As shown in Figure 4 The electronic device 10 can include a processor 100, a memory 101, a bus 102 and a communication interface 103, the processor 100, the communication interface 103 and the memory 101 are connected through the bus 102; the memory 101 stores a computer program executable on the processor 100, and the processor 100 executes the computer program to implement the method provided by any of the preceding embodiments of the present application.

[0117] The memory 101 can include a high-speed random access memory (RAM: Random Access Memory), and can also include a non-volatile memory, such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 103 (which can be wired or wireless), and the Internet, a wide area network, a local network, a metropolitan area network, etc. can be used.

[0118] The bus 102 can be an ISA bus, a PCI bus or an EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. The memory 101 is used to store programs, and the processor 100 executes the programs after receiving execution instructions. The method disclosed in any of the preceding embodiments of the present application can be applied to the processor 100 or implemented by the processor 100.

[0119] The processor 100 can be an integrated circuit chip with signal processing capability. In the implementation process, each step of the above method can be completed by the integrated logic circuit of hardware in the processor 100 or the instruction in the form of software. The processor 100 described above can be a general processor, which can include a central processing unit (CPU), a network processor (NP), etc.; or can be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a ready programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. Each method, step and logic block diagram disclosed in the embodiments of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can be any conventional processor. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as a hardware code processor for execution, or a combination of hardware and software modules in the code processor for execution. The software module can be located in a random access memory, a flash memory, a read only memory, a programmable read only memory or an electrically erasable programmable memory, a register or other mature storage medium in the art. The storage medium is located in the memory 101, and the processor 100 reads the information in the memory 101 and combines the hardware to complete the steps of the above method.

[0120] The electronic device provided by the embodiments of the present application and the method provided by the embodiments of the present application have the same beneficial effects as the method adopted, run or implemented by them.

[0121] Another embodiment of the present application provides a computer readable storage medium having a computer program stored thereon, which is executed by a processor to implement the point cloud super-resolution method of any of the above embodiments.

[0122] The embodiments of the present application also provide a computer readable storage medium corresponding to the method provided by the above-mentioned embodiments. Please refer to Figure 5 The computer readable storage medium shown is an optical disc 20, which has a computer program (i.e. program product) stored thereon, and the computer program will execute the method provided by any of the above embodiments when run by a processor.

[0123] It should be noted that examples of the computer readable storage medium can also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other optical, magnetic storage medium, which will not be described one by one here.

[0124] The computer readable storage medium provided by the above embodiments of the present application has the same inventive concept as the method provided by the embodiments of the present application, and has the same beneficial effects as the method adopted, run or implemented by the application stored therein.

[0125] It should be noted that:

[0126] The term "module" is not intended to be limited to a particular physical form. Depending on the specific application, a module can be implemented as hardware, firmware, software, and / or a combination thereof. In addition, different modules can share common components or even be implemented by the same components. There can or can not be a clear boundary between different modules.

[0127] The algorithms and displays presented herein are not inherently related to any particular computer, virtual apparatus, or other apparatus. Various general purpose systems can be used with these examples based upon the description herein. General purpose systems can be constructed to implement the required structure for the apparatuses described above using the description provided above. In addition, the present application is not described with reference to any particular programming language. It will be appreciated that a variety of programming languages can be used to implement the present application as described herein, and any references above to specific languages are provided for disclosure of enablement only.

[0128] It should be understood that although the steps in the flowcharts of the drawings are shown in a sequential order, such steps can not necessarily be performed in the order illustrated by the arrows. Unless otherwise specifically noted, the execution of the steps is not strictly limited to the order illustrated by the arrows, and can be performed in other orders. Moreover, at least some of the steps in the flowcharts of the drawings can include multiple sub-steps or multiple stages, which are not necessarily performed at the same time, but can be performed at different times, and the order of execution is not necessarily sequential, but can be performed in rotation or alternation with at least some of the steps or sub-steps or stages of other steps.

[0129] The above embodiments only express the implementation of the present application, and the description is more specific and detailed, but it should not be understood as limiting the scope of the patent of the present application. It should be noted that for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of protection of the present application. Therefore, the scope of protection of the present application should be subject to the appended claims.

Claims

1. A point cloud super-resolution method, characterized in that, include: An initial super-resolution model for point clouds is constructed and trained to obtain a point cloud super-resolution model that meets the accuracy threshold; the initial super-resolution model for point clouds includes a generator and a discriminator; The point cloud to be processed is divided into segmented point clouds, and the segmented point clouds are input into the point cloud super-resolution model; the point cloud coordinates and color information of the segmented point clouds are both over-normalized; The generator extracts the geometric and color information features of the segmented point cloud, and then uses the geometric and color information features to reconstruct the super-resolution point cloud. The discriminator determines whether the reconstructed super-resolution point cloud reaches a preset confidence threshold and outputs the super-resolution point cloud that reaches the preset confidence threshold. The process of dividing the point cloud to be processed into segmented point clouds includes: The point cloud to be processed is uniformly divided into small cubes of the same volume. Sampling is performed on the small cubes as units. During sampling, points of adjacent cubes are collected with a preset overlap rate to obtain the segmented point cloud. The training of the initial super-resolution model of the point cloud includes: using a composite loss as part of the training loss for the super-resolution task to train the initial super-resolution model of the point cloud; wherein the composite loss includes adversarial loss, shape-aware loss and color loss based on geometric location; The shape-aware loss includes global constraints and detail constraints; the global constraints are based on the distance between the two data distributions, the predicted point cloud and the real point cloud, calculated using EMD; the detail constraints are based on the average distance between point-to-point pairs between the predicted point cloud and the real point cloud, as well as the density metric of the generated point cloud.

2. The point cloud super-resolution method according to claim 1, characterized in that, The generator includes a parallel point cloud coordinate generation channel and a point cloud color generation channel; the point cloud coordinate generation channel is used to extract the geometric information features of the segmented point cloud; the point cloud color generation channel is used to extract the color information features of the segmented point cloud.

3. The point cloud super-resolution method according to claim 2, characterized in that, The initial super-resolution model for the training point cloud includes: The point cloud in the training set is input into the generator, and the point cloud coordinate generation channel and the point cloud color generation channel are upsampled twice consecutively to obtain the predicted point cloud. Calculate the prediction error of the predicted point cloud, update the parameters in the generator in reverse according to the prediction error, iterate until the prediction error reaches a preset threshold, and obtain the generator after the first training is completed. The point cloud in the training set and the corresponding predicted point cloud are input into the discriminator to determine the confidence level of the predicted point cloud. The initial super-resolution model of the entire point cloud is updated based on the confidence level determination result, and the process is iterated until the confidence level reaches a preset confidence threshold.

4. The point cloud super-resolution method according to claim 1, characterized in that, The step of extracting geometric and color information features of the segmented point cloud through the generator includes: The generator uses graph convolution operations based on k-nearest neighbors in three-dimensional space to extract geometric and color information features from the segmented point cloud.

5. A point cloud super-resolution device, characterized in that, include: A training module is constructed to build and train an initial super-resolution model of the point cloud, resulting in a point cloud super-resolution model that meets the accuracy threshold. The initial super-resolution model of the point cloud includes a generator and a discriminator; The segmentation module is used to divide the point cloud to be processed into segmented point clouds and input the segmented point clouds into the point cloud super-resolution model; the point cloud coordinates and color information of the segmented point clouds are both normalized. The reconstruction module is used to extract the geometric and color information features of the segmented point cloud through the generator, and the generator uses the geometric and color information features to reconstruct the super-resolution point cloud. The judgment output module is used to determine whether the reconstructed super-resolution point cloud reaches a preset confidence threshold through the discriminator, and output the super-resolution point cloud that reaches the preset confidence threshold. The segmentation module is also used to uniformly divide the point cloud to be processed into small cubes of the same volume, and to perform patch sampling with the small cubes as units. During sampling, points of adjacent cubes are collected with a preset overlap rate to obtain the patch point cloud. The training of the initial super-resolution model of the point cloud includes: using a composite loss as part of the training loss for the super-resolution task to train the initial super-resolution model of the point cloud; wherein the composite loss includes adversarial loss, shape-aware loss and color loss based on geometric location; The shape-aware loss includes global constraints and detail constraints; the global constraints are based on the distance between the two data distributions, the predicted point cloud and the real point cloud, calculated using EMD; the detail constraints are based on the average distance between point-to-point pairs between the predicted point cloud and the real point cloud, as well as the density metric of the generated point cloud.

6. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the program to implement the method as described in any one of claims 1-4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the method as described in any one of claims 1-4.

Images