Image processing method and related device

Abstract

The invention discloses an image processing method and a related device, and the method comprises the steps: obtaining a two-dimensional image of a preset object when the three-dimensional reconstruction of the preset object is needed, carrying out the feature extraction of the two-dimensional image, and obtaining the image features of the two-dimensional image. And performing feature conversion on the image features through a point cloud feature conversion network to obtain corresponding point cloud features, inputting the point cloud features into a model creation module, and outputting a three-dimensional model of the preset object based on the point cloud features through the model creation module. The point cloud features can reflect complete three-dimensional space information of the preset object and reflect the three-dimensional geometrical shape of the surface of the preset object, richer content is provided for three-dimensional reconstruction, even if some parts are invisible, information of the invisible areas can be accurately speculated and supplemented through the point cloud features of other angles, and the method is suitable for the three-dimensional reconstruction of the preset object. Therefore, the reconstruction effect of the three-dimensional model is improved, and the problems of serious geometric deformation, singularity or geometric fuzziness and the like are avoided.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to an image processing method and related apparatus. Background Technology

[0002] 3D reconstruction is a technique that processes, calculates, and reconstructs 3D images of an object to obtain its 3D reconstruction data, ultimately creating a realistic 3D model of the object in a computer. 3D reconstruction technology is a key technology for creating virtual reality representations of the real world in computers and has important applications in various fields.

[0003] Currently, 3D reconstruction is mainly performed using the Large Reconstruction Model (LRM) method. In the LRM method, image features can be extracted from the 2D image of the object, and then the 3D model of the object can be obtained by 3D reconstruction based on the image features.

[0004] However, this method can obtain relatively good 3D models for object regions visible in 2D images, but for object regions not visible in 2D images, the 3D models of objects are often poor, and serious geometric deformation, singularity, or geometric blur will occur. Summary of the Invention

[0005] To address the aforementioned technical problems, this application provides an image processing method and related apparatus that can improve the 3D reconstruction effect of 3D models and avoid serious geometric deformation, singularity, or geometric blurring.

[0006] The embodiments of this application disclose the following technical solutions:

[0007] On one hand, embodiments of this application provide an image processing method, the method comprising:

[0008] Obtain a two-dimensional image of a preset object;

[0009] Feature extraction is performed on the two-dimensional image to obtain the image features of the two-dimensional image;

[0010] The image features are transformed by a point cloud feature transformation network to obtain corresponding point cloud features. The point cloud features are used to represent the three-dimensional spatial information of the preset object. The point cloud feature transformation network is trained with standard point cloud features as supervision signals and sample images of sample objects as input. The trained point cloud feature transformation network has the function of converting image features into point cloud features. The point cloud features obtained by feature transformation have the same feature dimension as the standard point cloud features.

[0011] The point cloud features are input into the model creation module, and the model creation module outputs a 3D model of the preset object based on the point cloud features.

[0012] On one hand, embodiments of this application provide an image processing apparatus, the apparatus comprising an acquisition unit, an extraction unit, a conversion unit, and a generation unit:

[0013] The acquisition unit is used to acquire a two-dimensional image of a preset object;

[0014] The extraction unit is used to extract features from the two-dimensional image to obtain the image features of the two-dimensional image;

[0015] The conversion unit is used to perform feature conversion on the image features through a point cloud feature conversion network to obtain corresponding point cloud features. The point cloud features are used to represent the three-dimensional spatial information of the preset object. The point cloud feature conversion network is trained by using standard point cloud features as supervision signals and sample images of sample objects as inputs. The trained point cloud feature conversion network has the function of converting image features into point cloud features. The point cloud features obtained through feature conversion have the same feature dimension as the standard point cloud features.

[0016] The generation unit is used to input the point cloud features into the model creation module, and the model creation module outputs a 3D model of the preset object based on the point cloud features.

[0017] On one hand, embodiments of this application provide a computer device, the computer device including a processor and a memory:

[0018] The memory is used to store computer programs and to transfer the computer programs to the processor;

[0019] The processor is configured to execute the method described in any of the foregoing aspects according to instructions in the computer program.

[0020] In one aspect, embodiments of this application provide a computer-readable storage medium for storing a computer program that, when executed by a processor, causes the processor to perform the methods described in any of the foregoing aspects.

[0021] On one hand, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the method described in any of the foregoing aspects.

[0022] As can be seen from the above technical solutions, when it is necessary to perform 3D reconstruction of a preset object, a 2D image of the preset object can be obtained, and feature extraction can be performed on the 2D image to obtain the image features of the 2D image. However, this application does not directly perform 3D reconstruction based on image features, but instead uses a point cloud feature conversion network to perform feature conversion on the image features to obtain the corresponding point cloud features. The point cloud feature conversion network is trained using standard point cloud features, i.e., real point cloud features, as the supervision signal and sample images of the sample object as input. During the training process, the point cloud feature conversion network learns the conversion relationship between image features and point cloud features, enabling the trained point cloud feature conversion network to have the function of converting image features into point cloud features. Therefore, point cloud features can be accurately obtained based on image features through the point cloud feature conversion network. Point cloud features can represent the complete three-dimensional spatial information of a preset object and reflect the three-dimensional geometry of the object's surface. They can be used to understand and reconstruct the object from multiple angles and levels, providing richer content for 3D reconstruction. Thus, when the model creation module outputs a 3D model of a preset object based on point cloud features, even if some parts are not visible, the information of these invisible areas can be accurately inferred and supplemented through point cloud features from other angles, thereby improving the reconstruction effect of the 3D model and avoiding serious geometric deformation, singularity, or geometric blurring. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 An overall architecture diagram for 3D reconstruction provided for related technologies;

[0025] Figure 2 An application scenario architecture diagram of an image processing method provided in this application embodiment;

[0026] Figure 3 A flowchart illustrating an image processing method provided in an embodiment of this application;

[0027] Figure 4 This is a schematic diagram of the network structure of the diffusion model provided in an embodiment of this application;

[0028] Figure 5a A schematic diagram illustrating the principle of a Stable Diffusion model provided in this application embodiment;

[0029] Figure 5bA schematic diagram illustrating the principle of another Stable Diffusion model provided for embodiments of this application;

[0030] Figure 6 An example diagram illustrating how a single multi-view image is obtained through an image generation model, as provided in this application embodiment;

[0031] Figure 7 An example diagram illustrating a 3D model obtained by 3D reconstruction based on a 2D image, provided as an embodiment of this application;

[0032] Figure 8 An example diagram of a three-plane feature transformation provided in an embodiment of this application;

[0033] Figure 9 A flowchart illustrating a training method for a point cloud feature transformation network provided in an embodiment of this application;

[0034] Figure 10 This application provides an overall technical framework diagram of an image processing method according to an embodiment of the present application.

[0035] Figure 11 A structural diagram of an image processing apparatus provided in an embodiment of this application;

[0036] Figure 12 A structural diagram of a terminal provided in an embodiment of this application;

[0037] Figure 13 This is a structural diagram of a server provided in an embodiment of this application. Detailed Implementation

[0038] The embodiments of this application will now be described with reference to the accompanying drawings.

[0039] The relevant technology can perform 3D reconstruction using the LRM method. For details on the LRM method, please refer to [link to relevant documentation]. Figure 1 As shown, the LRM method takes a single 2D image of the object as input to achieve 3D object reconstruction based on the 2D image. This method first takes a 2D image of the object as input, extracts image features through an image encoder, then encodes the image features and the corresponding camera pose of the 2D image through an image-to-triplane decoder to obtain the corresponding triplane features. Finally, using the triplane features, a 3D model of the object is obtained through Neural Radiance Field (NeRF) reconstruction.

[0040] However, this method uses a single 2D image as input for 3D reconstruction of an object. The object region visible in the 2D image can obtain a relatively good 3D model, while the object region not visible in the 2D image often has a poor 3D model, and serious geometric deformation, singularity or geometric blur will occur.

[0041] To address the aforementioned technical problems, this application provides an image processing method. This method does not directly reconstruct 3D based on image features, but instead uses a point cloud feature transformation network to transform image features and obtain corresponding point cloud features. Point cloud features can represent the complete 3D spatial information of a preset object, reflecting the 3D geometric shape of the object's surface. This allows for understanding and reconstruction of the object from multiple angles and levels, providing richer content for 3D reconstruction. Thus, when the model creation module outputs a 3D model of the preset object based on point cloud features, even if some parts are invisible, information about these invisible areas can be accurately inferred and supplemented using point cloud features from other angles. This improves the reconstruction effect of the 3D model and avoids serious geometric deformation, singularities, or geometric blurring.

[0042] It should be noted that the image processing method provided in this application can be applied to various fields requiring 3D reconstruction, such as gaming, generative artificial intelligence (AIGC), extended reality (XR), smart homes, cultural relic reconstruction, audio-visual entertainment, virtual humans, digital humans, digital twins, and smart cities. These fields often require massive amounts of 3D model assets, which can be obtained through 3D reconstruction using the method provided in this application.

[0043] The image processing method provided in this application can be executed by a computer device, such as a server or a terminal. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. Terminals include, but are not limited to, smartphones, computers, smart voice interaction devices, smart home appliances, vehicle terminals, aircraft, XR devices, etc.

[0044] XR is a general term for Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR). By integrating the visual interaction technologies of these three, it brings users a sense of immersion that allows for seamless transition between the virtual and real worlds. Therefore, the XR device involved in this application can be a virtual reality device such as a VR device, AR device, or MR device.

[0045] like Figure 2 As shown, Figure 2 An application scenario architecture diagram of an image processing method is shown, which may include a server 200.

[0046] When a 3D reconstruction of a preset object is required, server 200 can acquire a 2D image of the preset object, extract features from the 2D image, and obtain image features of the 2D image. However, in this embodiment, server 200 does not directly perform 3D reconstruction based on image features. Instead, it performs feature transformation on the image features through a point cloud feature transformation network to obtain corresponding point cloud features. Then, it inputs the point cloud features into the model creation module, and the model creation module outputs a 3D model of the preset object based on the point cloud features, thereby realizing the 3D reconstruction of the preset object.

[0047] The point cloud feature conversion network is trained using standard point cloud features as supervision signals and sample images of the target object as input. This allows the network to learn the conversion relationship between image features and point cloud features during training, enabling it to convert image features into point cloud features. Therefore, the server 200 can accurately obtain point cloud features based on image features through the point cloud feature conversion network. Point cloud features can represent the complete three-dimensional spatial information of the target object, reflecting the three-dimensional geometry of its surface. They allow for understanding and reconstructing the object from multiple angles and levels, providing richer content for 3D reconstruction. Thus, when the model creation module outputs a 3D model of the target object based on point cloud features, even if some parts are invisible, information about these invisible areas can be accurately inferred and supplemented using point cloud features from other angles. This improves the reconstruction effect of the 3D model and avoids serious geometric deformation, singularities, or geometric blurring.

[0048] It should be noted that the methods provided in this application's embodiments may involve artificial intelligence (AI) technology. AI is the theory, methods, technology, and application systems that utilize digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine capable of reacting in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess perception, reasoning, and decision-making capabilities.

[0049] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, pre-trained model technology, operating / interactive systems, and mechatronics. Among these, pre-trained models, also known as large-scale models or foundational models, can be widely applied to downstream tasks across various AI fields after fine-tuning. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0050] With the research and advancement of artificial intelligence (AI) technology, AI is being studied and applied in various fields, such as smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, autonomous driving, drones, digital twins, virtual humans, robots, AI-generated content (AIGC), conversational interaction, smart healthcare, smart customer service, and game AI. It is believed that with the development of technology, AI will be applied in more fields and play an increasingly important role.

[0051] Specifically, the embodiments of this application may relate to computer vision technology within AI technology. Computer vision (CV) is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes for target recognition and measurement, and further performs image processing to create images more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision researches related theories and technologies, attempting to establish artificial intelligence systems capable of extracting information from images or multidimensional data. Large model technology has brought significant changes to the development of computer vision technology. Pre-trained models in the field of vision, such as the Swing Transformer model, the Vision Transformer (ViT) model, the Mixture of Experts (V-MOE) network, and Masked Autoencoders (MAE), can be quickly and widely applied to downstream specific tasks after fine-tuning. Computer vision technology typically includes image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, three-dimensional (3D) technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and other technologies. In this embodiment, computer vision technology can be used to reconstruct a 3D image of a preset object from a 2D image.

[0052] This application's embodiments may relate to machine learning in AI technology. Machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory, and many other disciplines. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and pre-trained learning. Pre-trained models are the latest development in deep learning, integrating the above techniques. In this application's embodiments, a point cloud feature transformation network can be obtained through machine learning training.

[0053] It should be noted that in the specific implementation of this application, the entire process may involve user information and other related data. When the above embodiments of this application are applied to specific products or technologies, separate consent or permission from the user is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0054] Next, the image processing method provided in the embodiments of this application will be described in conjunction with the accompanying drawings. This method can be executed by a computer device. See also Figure 3 , Figure 3 A flowchart of an image processing method is shown, which may include steps S301-S304, as detailed below:

[0055] S301. Obtain a two-dimensional image of a preset object.

[0056] 3D reconstruction is a technique that processes, calculates, and reconstructs a 3D image of an object to obtain its 3D reconstruction data. This data is then used to realistically reconstruct a 3D model of the object in a computer, allowing the object to be displayed as a 3D model in a virtual world. When 3D reconstruction of a preset object is needed, the computer can acquire a 2D image of that object.

[0057] The preset object can refer to the object that needs to be reconstructed in three dimensions. The object can be a real-world object, scene, etc. For example, an object can be a car, an animal, a handheld tool, etc., and a scene can be a video conferencing scene, a game scene, etc. This application does not limit this.

[0058] The two-dimensional image of the preset object can be an image obtained from the preset object itself. For example, if the preset object is a handheld tool, then the two-dimensional image of the preset object can be an image of the handheld tool. This application does not limit the method of obtaining the two-dimensional image of the preset object. In one possible implementation, the method of obtaining the two-dimensional image of the preset object can be to directly obtain a single image of the preset object. This single image can be an image obtained directly by taking a picture of the preset object, or it can be obtained by taking a screenshot. For example, if the preset object is a virtual object (such as a virtual character in a game), then the two-dimensional image of the preset object can be obtained by taking a screenshot of a scene including the virtual object from a game application.

[0059] In some cases, a 2D image can be an image of a predefined object viewed from a single perspective. However, 3D reconstruction requires understanding the 3D geometry of the object's surface, necessitating a multi-angle understanding to provide more comprehensive scene information for the reconstruction. Therefore, in another possible implementation, obtaining the 2D image of the predefined object can be achieved by acquiring an initial image of the object—a single image of the object—and then using an image generation model to generate multi-view images of the object from the initial image. This results in 2D images of the object from each of the multiple perspectives, which are then used to define the 2D images of the object as the 2D images of the object itself.

[0060] In this process, a single image can be obtained by directly photographing the preset object or by taking a screenshot. For example, if the preset object is a virtual object (such as a virtual character in a game), then the 2D image of the preset object can be obtained by taking a screenshot of the scene including the virtual object from a game application. The image generation model can be a model that generates 2D images from multiple perspectives based on a single image, and is trained to achieve this. Multi-view image generation can be a technique or method for generating 2D images from multiple different perspectives, where the perspective can be the angle from which the user views the preset object. This generation method can provide more comprehensive and richer information, which helps to understand and analyze the preset object more deeply. In multi-view image generation, specific algorithms or models are usually required to simulate the observation effects under different perspectives. These algorithms or models can capture and represent the appearance, shape, structure, and other features of the preset object under different perspectives, thereby generating multiple 2D images with different perspectives.

[0061] To obtain an image generation model capable of generating two-dimensional images corresponding to multiple viewpoints, the model can be pre-trained. The training process involves acquiring training samples, including training sample images and real images of those training sample images from different viewpoints. The model to be trained then outputs predicted images of the training sample images from different viewpoints based on the training sample images. Furthermore, the model parameters are adjusted according to the differences between the predicted and real images at each viewpoint, resulting in the image generation model capable of generating images from multiple viewpoints based on a single image.

[0062] It should be noted that the network structure of the image generation model is not limited in the embodiments of this application. In one possible implementation, the image generation model can be a stable diffusion model. A diffusion model is a model used to generate images, its main characteristic being the generation of images through gradual diffusion and iteration. The network structure of the diffusion model can be found in [reference needed]. Figure 4As shown, generating an image using a stable diffusion model involves two processes: forward diffusion (i.e., the diffusion process) and backward diffusion. The core of forward diffusion is training a denoising network (DenoisingUNet), which involves processing the original image (in...)... Figure 4 The algorithm adds noise to a base image (represented as X), using the noise as a label. A noisy image and a time step are input to the UNet, allowing it to predict the noise. Through continuous learning, the loss value between the predicted and actual noise is reduced, ultimately resulting in a UNet capable of recognizing noise. Backdiffusion, or the restoration process, involves inputting a noisy image and a time step into the model after training the UNet, iteratively removing the noise predicted by the UNet model to reconstruct the original image as much as possible. UNet is a convolutional neural network specifically designed for image segmentation tasks, named for its U-shaped structure. In this embodiment, the initial image is used as the control condition for the Stable Diffusion model, thereby generating multi-view images based on the initial image. The Stable Diffusion model outputs two-dimensional images of a preset object from multiple viewpoints (in...). Figure 4 (represented as Y in the original text).

[0063] Specifically, the principle of the Stable Diffusion model is as follows: Figure 5a and Figure 5b As shown, in Figure 5a and Figure 5b The random noise image x3 is denoised multiple times to obtain the original image x0. Each denoising process is processed by the Stable Diffusion model UNet network. The original image x0 is denoised multiple times to obtain the noisy image x3. q(x2|x1) represents the probability of obtaining x2 by adding noise to x1, q(x2|x1,x0) represents the probability of obtaining x1 by adding noise to x0 and obtaining x2 by adding noise to x1, and q(x3|x2,x0) represents the probability of obtaining x2 by adding noise to x0 and obtaining x3 by adding noise to x2. During training, the original image x0 can be input and different levels of noise can be superimposed to obtain xt (as shown in formula (1)). UNet can then attempt to denoise (i.e., recover the noise). After inputting xt into the UNet network, the predicted noise is obtained. The error between the predicted noise and the true noise value is calculated, and the loss function is obtained. The UNet network is then trained based on the loss function to obtain the correct denoising ability, as shown in formula (2). After training the UNet network, various noises xt can be generated during inference, and the original image x0 is obtained after multiple UNet denoising processes. This realizes the generative capability of the Stable Diffusion model.

[0064]

[0065] Where x0 represents the original image, x t Represents a noisy image, q(x) t |x0) represents generating x by adding noise to x0. t The probability, x 1:t Represents x1 to x t x 1:t-1 Indicates x1 to Let α represent a Gaussian distribution. t Let represent the hyperparameter, and I represent the (0,1) normal distribution.

[0066]

[0067] Among them, L γ (∈ θ ) represents the loss function, γ t α t This represents the hyperparameter, and T represents the number of times noise is added. Represents the UNet network in the stable diffusion model, ∈ t Represents the true value of the noise. Let x0 represent the original image, and x0~q(x0) represent the prediction noise. Represents ∈ t Follows a Gaussian distribution. It expresses expectation.

[0068] Based on the above principle, when the image generation model is a Stable Diffusion model, in the embodiments of this application, when generating two-dimensional images corresponding to multiple viewpoints using the image generation model, the initial image can be input into the Stable Diffusion model, and then the two-dimensional images of the preset object corresponding to multiple viewpoints can be obtained through the Stable Diffusion model.

[0069] By generating 2D images of a preset object from multiple perspectives, more comprehensive and richer information can be provided, facilitating a deeper understanding and analysis of the object. This allows for improved 3D reconstruction results based on the 2D images from different perspectives. Furthermore, the multiple perspectives can differ, enabling 3D reconstruction based on different 2D images, thus providing designers with creative inspiration.

[0070] It should be noted that when generating multi-view images based on the image generation module, the image generation model can output two-dimensional images from each viewpoint. In one possible implementation, to ensure the consistency of the two-dimensional images corresponding to multiple viewpoints, the image generation model can also output a single image that includes the combination of views corresponding to multiple viewpoints. In this case, the way to obtain the two-dimensional images of the preset object corresponding to multiple viewpoints by generating multi-view images from the initial image through the image generation model can be to generate multi-view images from the initial image through the image generation model to obtain the multi-view images corresponding to the initial image. These multi-view images are single images obtained by the combination of views corresponding to multiple viewpoints. Then, the two-dimensional images of the preset object in each viewpoint of the multiple viewpoints are cropped from the multi-view images.

[0071] The process of obtaining a single multi-view image through an image generation model can be found in [link to image generation model]. Figure 6 As shown, Figure 6 Taking a handheld tool as an example, the dimensions of the two-dimensional image of this preset object are W×H, such as... Figure 6 As shown in Figure 601, the output multi-view image has a size of mW×nH, where m×n represents the number of generated multiple views, m represents the number of rows in the view corresponding to each view, and n represents the number of columns in the view corresponding to each view. The obtained multi-view image can be represented as I_multi, as shown in Figure 601. Figure 6 As shown in Figure 602, the multi-view image in Figure 602 includes multiple views corresponding to different perspectives. After obtaining the multi-view image shown in Figure 602, a two-dimensional image for each perspective can be cropped from the entire multi-view image. The two-dimensional image for each perspective can be represented as I. k This results in m×n two-dimensional images corresponding to m×n viewpoints. Figure 6 In this case, m is 3, n is 2, and 6 two-dimensional images are laid out to form a single image with a 3×2 layout.

[0072] The above method of generating multi-view images takes into account the mutual communication between views from different perspectives, and generates multiple views in a single two-dimensional image, thereby ensuring the consistency of multiple views in the multi-view image, which greatly improves the downstream three-dimensional reconstruction effect.

[0073] Understandably, in some cases, multiple viewpoints can be set based on actual needs, allowing control over which viewpoints the image generation model generates 2D images. In this scenario, the method of generating 2D images of a preset object from an initial image using the image generation model, corresponding to different viewpoints, can be to obtain control conditions. These control conditions indicate multiple viewpoints, and then, according to the multiple viewpoints indicated by the control conditions, control the image generation model to generate 2D images of the preset object corresponding to different viewpoints.

[0074] The control conditions can be the conditions that control the image generation model to generate two-dimensional images corresponding to multiple viewpoints based on the initial image. In this embodiment, the control conditions can be multiple viewpoints. Assuming the viewpoint corresponding to the initial image is the current viewpoint, the multiple viewpoints can be viewpoints obtained by rotating the current viewpoint along a certain direction (clockwise or counterclockwise) by different angles, such as rotating it clockwise by 30 degrees, 60 degrees, 90 degrees, 120 degrees, 150 degrees, and 180 degrees in sequence, thereby obtaining multiple viewpoints as control conditions to generate two-dimensional images corresponding to the multiple viewpoints.

[0075] By using the input angle as a control condition, the image generation model can receive explicit instructions about the desired viewpoint for generating the image and generate an image from the corresponding viewpoint, thereby helping to ensure that the generated image meets the needs of the user or application.

[0076] S302. Perform feature extraction on the two-dimensional image to obtain the image features of the two-dimensional image.

[0077] After obtaining a two-dimensional image of a preset object, feature extraction can be performed on the two-dimensional image to obtain its image features. Feature extraction is a key step in computer vision and machine learning, referring to the process of extracting meaningful, representative, and computationally tractable information (i.e., "features") from raw data. These features are typically used to describe the attributes or characteristics of the data for further analysis or model training. The purpose of feature extraction is to transform raw data into more representative and interpretable features to better describe and differentiate the data. In this embodiment, the raw data may refer to a two-dimensional image of a preset object. The image features may be representative and interpretable features extracted from the two-dimensional image of the preset object, used in subsequent three-dimensional reconstruction steps.

[0078] In this application embodiment, feature extraction can be performed using an image encoder. The image encoder used in this application embodiment can be obtained through training. This application embodiment does not limit the network structure of the image encoder; the image encoder can be any encoder capable of feature extraction, such as the Contrastive Language-Image Pre-Training (CLIP) model or the Dino feature model. The CLIP model is a pre-trained model used to extract features from matched images and text. For a two-dimensional image and its corresponding text description, the extracted image features are typically one-dimensional feature vectors, and the text features are also one-dimensional vectors with dimensions of 512, 768, 1024, etc. This application can utilize the CLIP model to extract features from two-dimensional images, obtaining one-dimensional feature vectors as image features. The Dino feature model is an image feature extraction model that generally uses a self-supervised vision transformer (ViT) to extract image features. It has good generalization ability and can achieve high accuracy in multiple computer vision tasks.

[0079] The expression for feature extraction using an image encoder can be shown below:

[0080] f_image = Image_encoder(I)

[0081] Where f_image represents the image features of a two-dimensional image, Image_encoder represents the corresponding image encoder, and I represents the two-dimensional image of the preset object.

[0082] It is understandable that when the 2D image of the preset object consists of 2D images corresponding to multiple viewpoints, the method for extracting features from the 2D images to obtain image features can be to extract features from the 2D images at each viewpoint separately, thus obtaining image features for each viewpoint. The image features of the 2D images at each viewpoint can be represented as f image_k k represents the number of perspectives, such as 1, 2, 3, ..., m×n. For example, if the number of perspectives is m×n, then k is 1, 2, 3, ..., m×n.

[0083] When performing feature extraction using an image encoder, the expression for feature extraction of a 2D image at each viewpoint can be as follows:

[0084] f_image_k=Image_encoder(I_k)

[0085] Where f_image_k represents the image features of the 2D image from the k-th viewpoint, Image_encoder represents the corresponding image encoder, and I_k represents the 2D image from the k-th viewpoint.

[0086] S303. The image features are transformed using a point cloud feature transformation network to obtain the corresponding point cloud features.

[0087] After obtaining image features, these features can be input into a point cloud feature transformation network (PDN), which then converts the image features into corresponding point cloud features. The PDN is trained using standard point cloud features as supervision signals and sample images of the target object as input. Standard point cloud features are realistic and accurate point cloud features. These features can be obtained directly from point cloud data, extracted using accurate and reliable feature extraction methods, or through manual annotation. By training the PDN with realistic and accurate point cloud features, it learns the conversion relationship between image features and point cloud features, enabling it to convert image features into point cloud features. Therefore, the PDN can accurately obtain point cloud features based on image features, and the converted point cloud features have the same feature dimensions as the standard point cloud features. For example, if standard point cloud features include four dimensions (A, B, C, and D), the converted point cloud features should also include these four dimensions. In this embodiment, "same feature dimension" can mean that the converted point cloud features and the standard point cloud features have the same feature size and number of channels. Point cloud features are used to represent the three-dimensional spatial information of a preset object, and can represent the complete three-dimensional spatial information of the preset object, reflect the three-dimensional geometric shape of the surface of the preset object, and can understand the preset object from multiple angles and levels, providing richer content for three-dimensional reconstruction.

[0088] Point cloud feature transformation networks (PDVNs) are deep learning models that can be trained. The purpose of a DMVN is to convert image features in an image into point cloud features, achieving feature alignment between the image and point cloud data. In other words, feature alignment ensures a one-to-one correspondence between detected image features (such as edges, corners, and textures) and point cloud features. This correspondence allows for a better understanding of the structure and layout of a predefined object in three-dimensional space. In this embodiment, the image features processed by the DMVN are those in a two-dimensional image, and the resulting point cloud features are used for subsequent three-dimensional reconstruction.

[0089] The embodiments of this application do not limit the network structure of the point cloud feature transformation network. For example, it can be a network structure based on Transformer, which can be a neural network model based on self-attention mechanism.

[0090] When the image features are the image features of the two-dimensional image under each viewpoint, the point cloud features can be obtained by performing feature transformation on the image features through the point cloud feature transformation network. The point cloud features of the two-dimensional image under each viewpoint can be obtained by performing feature transformation on the image features of the two-dimensional image under each viewpoint through the point cloud feature transformation network, and then summing the point cloud features of the two-dimensional image under each viewpoint to obtain the point cloud features.

[0091] Taking image features represented by f_image_k as an example, the expression for converting the image features of a 2D image from each viewpoint into the corresponding point cloud features through a point cloud feature transformation network is as follows:

[0092] f_trans_k=Transformer(f_image_k)

[0093] Where f_trans_k represents the point cloud features under the k-th view after transformation, f_image_k represents the image features of the two-dimensional image under the k-th view, and Transformer() represents the point cloud feature transformation network based on the Transformer network structure.

[0094] The point cloud features obtained by feature transformation can be represented as:

[0095]

[0096] Where f_trans_all represents the point cloud features obtained by feature transformation, f_trans_k represents the point cloud features under the k-th view after transformation, and m×n represents the number of multiple viewpoints.

[0097] The above methods can capture point cloud features from multiple perspectives, thereby providing more comprehensive and richer information for analyzing the spatial structure of the preset object, so as to improve the effect of 3D reconstruction.

[0098] S304. Input the point cloud features into the model creation module, and output the three-dimensional model of the preset object based on the point cloud features through the model creation module.

[0099] After obtaining point cloud features through feature transformation, a 3D model of the preset object can be obtained through 3D reconstruction based on the obtained point cloud features. To this end, the obtained point cloud features can be input into the model creation module, which then outputs a 3D model of the preset object based on the point cloud features. This 3D model is a model created for the preset object in virtual 3D space. Point cloud features can reflect the complete 3D spatial information of the preset object, reflecting the 3D geometry of the object's surface. It allows for understanding and reconstructing the object from multiple angles and levels, providing richer content for 3D reconstruction. Even if some parts are occluded or invisible, information about these invisible areas can be accurately inferred and supplemented through point cloud features from other angles, thereby improving the reconstruction effect of the 3D model and avoiding serious geometric deformation, singularities, or geometric blurring.

[0100] The 3D model obtained from 3D reconstruction can be called a 3D mesh. In computer graphics and 3D modeling, "mesh" is a common term referring to the digital representation of a pre-defined 3D object composed of vertices, edges, and faces (usually triangles or quadrilaterals). This representation method allows for the storage, processing, and rendering of complex 3D shapes in a computer. 3D reconstruction based on point cloud features can begin by obtaining a surface model of the pre-defined object using a surface reconstruction algorithm, and then converting the surface model into a mesh model. The surface reconstruction algorithm can be Poisson Surface Reconstruction, Moving Least Squares (MLS), triangulation, etc., and the mesh can be, for example, a triangular mesh or a quadrilateral mesh.

[0101] See Figure 7 As shown, Figure 7 Taking a handheld tool as the preset object as an example, a two-dimensional image of the preset object can be obtained by photographing the handheld tool. Figure 7 The image shown in section 701 is a two-dimensional image of the preset object. After three-dimensional reconstruction through the aforementioned steps S301-S304, a three-dimensional model of the handheld tool can be obtained. The rendering of the obtained three-dimensional model can be found in [reference needed]. Figure 7 As shown in Figure 702. Figure 702 is merely an example of a 3D model in a 3D virtual space. It is a 3D model with a better effect obtained based on the 2D image shown in Figure 701. Due to limitations in drawing, the 3D effect of the 3D model is somewhat limited.

[0102] It should be noted that the main surfaces and edges of the preset object can more accurately represent its geometry and structure. This information is crucial for 3D reconstruction and helps to build a more accurate 3D model. The surfaces and edges of the preset object can be represented by three-plane features. Therefore, in one possible implementation, the model creation module can include a trained planar feature transformation network and a trained generative network. In step S304, point cloud features are input into the model creation module. The way the model creation module outputs the 3D model of the preset object based on the point cloud features could be by inputting the point cloud features into the planar feature transformation network, which performs three-plane feature transformation on the point cloud features in the 3D space formed by three orthogonal planes, obtaining three-plane features. The three-plane features represent the voxels to which the points corresponding to the point cloud features belong in 3D space. Then, the generative network outputs the 3D model of the preset object based on the three-plane features.

[0103] Triplane feature transformation refers to the transformation of features in 3D computer vision and image processing using three orthogonal planes (usually the XY, XZ, and YZ planes). Figure 8 This is a feature representation method that uses information from point cloud data (as shown) to describe points or regions in three-dimensional space. This feature representation method plays an important role in tasks such as processing point cloud data, 3D reconstruction, surface analysis, and rendering. A voxel can be the smallest unit for segmenting three-dimensional space, equivalent to a pixel in three-dimensional space. Here, the three-dimensional space can be the three-dimensional space composed of the three orthogonal planes mentioned above. By assigning features to each voxel through three-plane feature transformation, three-plane features are obtained, thus converting point cloud features into three-plane features. For example, in... Figure 8 In the process, point cloud features are converted to... Figure 8 The three-plane features of the voxel containing the black dot.

[0104] The three-plane feature transformation can be achieved through a planar feature transformation network, which can be trained. This planar feature transformation network can be a multi-layer deep learning convolutional layer or a transformer module; this embodiment does not limit this. The expression for converting point cloud features into three-plane features can be as follows:

[0105] f_pf1=Plane_feature(f_trans_all)

[0106] Where f_pf1 represents the transformed three-plane features, Plane_feature represents the plane feature transformation network, and f_trans_all represents the point cloud features.

[0107] Converting point cloud features into three-plane features allows for a more accurate description of the geometry and structure of a predefined object, enabling the construction of a more precise 3D model. Simultaneously, three-plane features organize point cloud features into a more manageable structure. By extracting and representing these features, predefined objects in a 3D scene can be described more concisely, reducing data redundancy and simplifying subsequent 3D reconstruction and analysis processes. This significantly reduces the amount of data that needs processing, thereby lowering computational complexity, improving processing efficiency, and making the 3D reconstruction process faster and more efficient, thus reducing asset production costs.

[0108] After obtaining the three-plane features, the generative network can perform 3D reconstruction of the preset object in different ways, such as NeRF, sign distance function (SDF), etc. This application mainly describes the SDF method. SDF, also known as oriented distance function, is a continuous function that maps a point p = (x, y, z) in 3D space to a real number s = SDF(p). The sign of s represents whether the point is inside or outside the surface of the preset object, and the absolute value of s represents the distance from the point to the surface of the preset object. A positive value indicates the point is inside the surface of the preset object, a negative value indicates the point is outside the surface, and a zero value indicates the point is on the surface. Based on the principle of SDF, in one possible implementation, the generative network outputs the 3D model of the preset object based on the three-plane features by calculating the first directed distance function value between the vertices of the voxels to which the three-plane features belong and the surface of the preset object. Then, the generative network extracts the vertices with a first directed distance function value of zero and uses these vertices to generate the 3D model. This process can be implemented using an SDF network, which can be trained.

[0109] When extracting vertices with a first directed distance function value of zero and generating a 3D model using these vertices, the corresponding 3D model can be calculated using methods such as Marching Cube (MC). Marching Cube is a classic algorithm in face rendering, using the Marching Cube algorithm to process the first directed distance function value. Specifically, the first directed distance function value can be divided into regular cubic units (voxels), and these voxels are processed one by one. Within each voxel, the surface condition inside the voxel is determined based on the relationship between the first directed distance function value and a certain threshold. Next, surface triangles are generated based on the surface condition, and the positions of the triangle vertices are calculated. This typically involves interpolation calculations to determine the intersection points of the isosurface and the edges of the cubic unit. Finally, the generated triangles are added to the final 3D model, thus obtaining the 3D model generated from the first directed distance function value.

[0110] SDF is a continuous function that can represent the distance from a given point (e.g., a vertex) to the surface of a preset object, as well as whether the vertex is inside (negative) or outside (positive) the preset object. This allows SDF to learn fully continuous shape functions of arbitrary precision, thereby achieving high-precision 3D reconstruction.

[0111] As can be seen from the above technical solutions, when it is necessary to perform 3D reconstruction of a preset object, a 2D image of the preset object can be obtained, and feature extraction can be performed on the 2D image to obtain the image features of the 2D image. However, this application does not directly perform 3D reconstruction based on image features, but instead uses a point cloud feature conversion network to perform feature conversion on the image features to obtain the corresponding point cloud features. The point cloud feature conversion network is trained using standard point cloud features, i.e., real point cloud features, as the supervision signal and sample images of the sample object as input. During the training process, the point cloud feature conversion network learns the conversion relationship between image features and point cloud features, enabling the trained point cloud feature conversion network to have the function of converting image features into point cloud features. Therefore, point cloud features can be accurately obtained based on image features through the point cloud feature conversion network. Point cloud features can represent the complete three-dimensional spatial information of a preset object and reflect the three-dimensional geometry of the object's surface. They can be used to understand and reconstruct the object from multiple angles and levels, providing richer content for 3D reconstruction. Thus, when the model creation module outputs a 3D model of a preset object based on point cloud features, even if some parts are not visible, the information of these invisible areas can be accurately inferred and supplemented through point cloud features from other angles, thereby improving the reconstruction effect of the 3D model and avoiding serious geometric deformation, singularity, or geometric blurring.

[0112] As described in the foregoing embodiments, the key to improving the reconstruction effect in this application is that after obtaining a two-dimensional image of the preset object, instead of directly performing three-dimensional reconstruction based on the image features of the two-dimensional image, a point cloud feature conversion network is used to convert the image features into point cloud features, and then three-dimensional reconstruction is performed based on the converted point cloud features. Therefore, the accuracy of the point cloud features can affect the reconstruction effect, and the accuracy of the point cloud features depends on the performance of the point cloud feature conversion network. The performance of the point cloud feature conversion network is determined by the model training process. The training method of the point cloud feature conversion network will be introduced next to obtain a point cloud feature conversion network with better performance.

[0113] See Figure 9 As shown, Figure 9 A flowchart illustrating a training method for a point cloud feature transformation network is shown. The method includes steps S901-S904, as detailed below:

[0114] S901. Obtain the sample image of the sample object and obtain the standard point cloud features.

[0115] To train a point cloud feature transformation network, training samples can be obtained. These training samples may include sample images of the sample objects and standard point cloud features. The standard point cloud features are ground truth values ​​during the training process and can be used as supervision signals for training the point cloud feature transformation network.

[0116] The sample object can refer to an object with a standard 3D model during the training process. The object can be a real-world or virtual object, scene, etc. For example, objects in the real world can be vehicles, animals, handheld tools, etc., and virtual objects in the virtual world can be virtual objects (such as game characters in games). Scenes can be video conferencing scenes, etc. This application does not limit this. The standard 3D model can be a known and accurate 3D model. The standard 3D model can be obtained by 3D reconstruction through accurate and reliable 3D reconstruction methods, or it can be obtained through manual annotation, etc.

[0117] The sample image of the sample object can be an image obtained specifically for the sample object. For example, if the sample object is a handheld tool, then the sample image of the sample object can be an image of the handheld tool. This application does not limit the method of obtaining the sample image of the sample object. In one possible implementation, the sample image of the sample object can be obtained by directly photographing the sample object.

[0118] In some cases, a sample image can be an image of a sample object viewed from a single perspective. However, 3D reconstruction requires understanding the 3D geometry of the sample object's surface, necessitating a multi-angle understanding to provide more comprehensive scene information for 3D reconstruction. Therefore, in another possible implementation, obtaining sample images of the sample object can be achieved by acquiring an initial sample image, which is a single image of the sample object. Then, an image generation model is used to generate multi-view images from the initial sample image, resulting in 2D images of the sample object corresponding to each of the multiple perspectives. These 2D images are then used to define the sample images of the sample object. The initial sample image can be a single image captured from the sample object or a screenshot. For example, if the sample object is a virtual object (such as a virtual character in a game), the initial sample image can be a screenshot from a game application that includes the virtual object.

[0119] The image generation model used during training can be found in [reference needed]. Figure 3 The corresponding embodiment, which describes how to generate two-dimensional images of a sample object from multiple viewpoints using an image generation model, can also be found in [reference needed]. Figure 3 The corresponding embodiments will not be described in detail here.

[0120] It should be noted that when generating multi-view images based on the image generation module, the image generation model can output sample images for each viewpoint separately. In one possible implementation, to ensure the consistency of sample images corresponding to multiple views, the image generation model can also output a single sample image consisting of a combination of views corresponding to each viewpoint. In this case, the method of generating multi-view images of the sample object from the initial sample image using the image generation model to obtain two-dimensional images of the sample object from multiple views can be as follows: The initial sample image is generated by the image generation model to obtain multi-view sample images corresponding to the initial sample image. These multi-view sample images are single sample images obtained by combining views corresponding to multiple views. Then, the two-dimensional images of the sample object from each viewpoint are cropped from the multi-view sample images. For details of this process, please refer to [link to relevant documentation]. Figure 3 The description of the corresponding embodiments will not be repeated here.

[0121] Standard point cloud features are true and accurate point cloud features that reflect the three-dimensional spatial information of a sample object. In one possible implementation, the point cloud features generated by a trained point cloud encoder can be defined as standard point cloud features. The trained point cloud encoder is trained on point cloud data and is a model capable of accurately extracting point cloud features. The point cloud features generated by the trained point cloud encoder are obtained by extracting features from the point cloud data of the sample object. Thus, accurate point cloud features are generated by the trained point cloud encoder, and these are defined as standard point cloud features. This accurate standard point cloud feature can then be used to train a high-performance point cloud feature transformation network.

[0122] Among them, a point cloud encoder can be a commonly used neural network structure for extracting point cloud features. A point cloud encoder can be any network that extracts features from point cloud data. Commonly used point cloud encoders include: PointNet, Point Bidirectional Encoder Representations from Transformers (PointBERT), Point Multilayer Perceptron (PointMLP), PointNext, etc. PointNext is the next version of PointNet. These neural network structures usually take point cloud data as input and output the feature vector corresponding to the point cloud data, that is, the point cloud features.

[0123] In one possible implementation, the expression for the point cloud features generated by the point cloud encoder can be as follows:

[0124] f_point = Encoder_3d(point)

[0125] Here, f_point represents the point cloud feature, Encoder_3d represents the point cloud encoder, and point represents the point cloud data. Therefore, a standard point cloud feature can be represented by f_point.

[0126] It is understandable that the trained point cloud encoder is obtained based on point cloud data. The point cloud encoder can be obtained during the training of a 3D generative network model using point cloud data of sample objects. The trained 3D generative network model includes the trained point cloud encoder and a model creation module trained based on the initial creation module. The process of training the 3D generative network model involves acquiring point cloud data of sample objects, then extracting features from the point cloud data using the point cloud encoder to obtain second predicted point cloud features. These second predicted point cloud features can be point cloud features obtained through feature extraction based on the point cloud data of the sample objects. These second predicted point cloud features are then input into the initial creation module, which outputs a predicted 3D model of the sample objects based on these features. This allows for adjustments to the point cloud encoder and the initial creation module based on the differences between the predicted 3D model and the standard 3D model of the sample objects, thus training the 3D generative network model. The 3D generative network model includes the trained point cloud encoder and the model creation module trained based on the initial creation module, thereby training the point cloud encoder.

[0127] Point cloud data of a sample object can be a discrete dataset composed of a series of three-dimensional coordinate points, used to describe the geometric features of the object such as surface shape, spatial location, and size. The main characteristics of point cloud data are high precision, high resolution, and high dimensionality of geometric information, which can intuitively represent the shape, surface, and texture of objects in space. Point cloud data is mainly generated through the following methods:

[0128] 1. Laser Scanner: A laser scanner is a common method for acquiring point cloud data. It utilizes the principle of laser ranging to scan the surface of an object (such as the surface of a sample object), acquiring a large number of coordinate points on the object's surface to form point cloud data.

[0129] 2. Structured light scanner: A structured light scanner calculates the three-dimensional coordinates of an object by projecting a known light pattern onto the object's surface and combining this with the light and shadow information obtained by a camera.

[0130] 3. Depth Camera: A depth camera is a method that can directly acquire depth information of an object's surface. Utilizing optical principles and combining the reflection information of the object's surface, it directly acquires the object's depth information and generates point cloud data.

[0131] 4. Robot visual navigation: The robot acquires images of its surrounding environment through visual sensors, and uses computer vision algorithms to extract and match image features to generate point cloud data for the robot's localization and navigation.

[0132] Feature extraction here is similar to that performed on two-dimensional images, except that it targets point cloud data of sample objects, which will not be elaborated further. The extracted point cloud features (e.g., second predicted point cloud features) can be used to describe the structure and shape of the point cloud, thereby reducing redundant information.

[0133] The initial creation module can be a model performing 3D reconstruction based on the features of the second predicted point cloud. The initial creation module can be an SDF network; this application does not limit this. For details, please refer to [link to relevant documentation]. Figure 3 The relevant descriptions of the corresponding embodiments will not be repeated here. The predicted 3D model can be the 3D model output by the 3D generative network model being trained during the 3D generative network model training process, after 3D reconstruction. The standard 3D model is a realistic and accurate 3D model of the sample object, which can be used to supervise the training of the 3D generative network model. Therefore, after obtaining the predicted 3D model, the differences between the predicted 3D model and the standard 3D model can be used to adjust the point cloud encoder and the initial creation module to train the 3D generative network model. For example, a reconstruction loss function can be constructed based on the predicted 3D model and the standard 3D model, and then the point cloud encoder and the initial creation module can be adjusted using the reconstruction loss function.

[0134] The above method trains a 3D generative network model using point cloud data of sample objects. Since the point cloud data of sample objects can reflect the complete 3D spatial information of sample objects, it can understand sample objects from multiple angles and levels, providing richer content for 3D reconstruction, thereby training a more accurate 3D generative network model (including a trained point cloud encoder), thus ensuring the acquisition of accurate standard point cloud features.

[0135] It should be noted that the main surfaces and edges of the sample object can more accurately represent its geometry and structure. This information is crucial for 3D reconstruction and helps to build a more accurate predicted 3D model. The surfaces and edges of the sample object can be represented by predicted three-plane features. Therefore, in one possible implementation, the initial creation module also includes a first initial network and a second initial network. The way the initial creation module outputs the predicted 3D model of the sample object based on the second predicted point cloud features can be by using the first initial network to perform three-plane feature transformation on the second predicted point cloud features in a 3D space composed of three orthogonal planes, obtaining predicted three-plane features. The predicted three-plane features represent the voxels to which the points corresponding to the second predicted point cloud features belong in 3D space. Then, the predicted three-plane features are input into the second initial network, which outputs the predicted 3D model based on the predicted three-plane features. The first initial network is the basic model framework used to train the planar feature transformation network, and can be any neural network model structure; this application embodiment does not limit this. The second initial network is the basic model framework used to train the generative network, and can be any neural network model structure; this application embodiment does not limit this.

[0136] The method of performing three-plane feature transformation during training and Figure 3 The method for performing three-plane feature transformation on point cloud features in the corresponding embodiment is similar and will not be repeated here. The expression for performing three-plane feature transformation on the second predicted point cloud features during training can be as follows:

[0137] f_pf2=Plane_feature(f_point)

[0138] Where f_pf2 represents the predicted three-plane features, Plane_feature represents the plane feature transformation network, and f_point represents the second predicted point cloud feature obtained by the point cloud encoder from the point cloud data feature extraction.

[0139] In the case where the 3D generative network model also includes a first initial network and a second initial network, the 3D generative network model can be trained by adjusting the point cloud encoder and the initial creation module using the differences between the predicted 3D model and the standard 3D model of the sample object. The 3D generative network model includes a trained point cloud encoder, a planar feature transformation network trained based on the first initial network, and a generative network trained based on the second initial network.

[0140] Converting the second predicted point cloud features into predicted three-plane features allows for a more accurate description of the geometry and structure of sample objects, enabling the construction of more precise predicted 3D models. Simultaneously, predicted three-plane features organize the second predicted point cloud features into a more manageable structure. By extracting and representing these features, sample objects in the 3D scene can be described more concisely, reducing data redundancy and simplifying subsequent 3D reconstruction and analysis processes. This significantly reduces the amount of data that needs to be processed, thereby lowering computational complexity, improving processing efficiency, and making the 3D reconstruction process faster and more efficient, thus reducing asset production costs.

[0141] After obtaining the predicted three-plane features, the preset object can be reconstructed in 3D using different methods, such as NeRF and signed distance function (SDF) methods. This application mainly describes the SDF method. In this case, the method of outputting the predicted 3D model based on the predicted three-plane features through the second initial network can be as follows: the second initial network calculates the second directed distance function value between the vertices of the voxels to which the predicted three-plane features belong and the surface of the sample object; then, the second initial network extracts the vertices with a second directed distance function value of zero, and uses the vertices with a second directed distance function value of zero to generate the predicted 3D model. For the specific implementation of 3D reconstruction using the SDF method during training, please refer to [link to relevant documentation]. Figure 3 The description of the corresponding embodiments will not be repeated here.

[0142] SDF is a continuous function that can represent the distance from a given point (e.g., a vertex) to the surface of a sample object, as well as whether the vertex is inside (negative) or outside (positive) the sample object. This allows SDF to learn fully continuous shape functions of arbitrary precision, thereby enabling the training of a generative network that can achieve high-precision 3D reconstruction.

[0143] S902. Extract features from the sample image to obtain the sample image features.

[0144] After obtaining the sample image, feature extraction can be performed on the sample image to obtain the sample image features. The implementation method for feature extraction of the sample image can be found in the feature extraction method described in S302. However, the feature extraction in S902 is for the sample image rather than the two-dimensional image, and the result is the sample image features rather than the image features.

[0145] S903. The features of the sample image are transformed through the initial network model to obtain the first predicted point cloud features.

[0146] The initial network model can be the basic model framework used to train the point cloud feature transformation network. The initial network model is used to perform feature transformation on the sample image features to obtain the first predicted point cloud features. The first predicted point cloud features can be point cloud features obtained by feature transformation of the sample image features; they are predicted values ​​obtained during training. The first predicted point cloud features can be represented by f_trans_all. S904. Based on the difference between the first predicted point cloud features and the standard point cloud features, the initial network model is trained to obtain the point cloud feature transformation network.

[0147] The first predicted point cloud feature is the predicted value during the training process, and the standard point cloud feature is the ground truth during the training process. Therefore, during the training of the point cloud feature transformation network, the initial network model can be trained based on the difference between the first predicted point cloud feature and the standard point cloud feature to obtain the point cloud feature transformation network.

[0148] During training, the extracted sample image features can be aligned with standard point cloud features. This typically involves steps such as feature matching and transformation matrix calculation. Using a feature matching algorithm, corresponding feature points in the point cloud data of the sample image and the sample object are found, and the transformation relationship between them is calculated. Then, based on these transformation relationships, the sample image features are transformed to the same coordinate system as the standard point cloud features, achieving feature alignment.

[0149] Then, the aligned standard point cloud features are used as supervision signals to train the initial network model. During training, the initial network model learns how to convert input image features into corresponding point cloud features, thus obtaining a point cloud feature conversion network that can accurately convert image features into point cloud features.

[0150] In this embodiment, standard point cloud features obtained from point cloud data are used as supervision signals. Standard point cloud features are real and accurate point cloud features. Using them as supervision signals enables the initial network model to continuously learn how to convert image features into accurate point cloud features, thereby training a point cloud feature conversion network that can accurately convert image features into point cloud features.

[0151] When training a point cloud feature transformation network based on the difference between the first predicted point cloud features and the standard point cloud features, a target loss function can be constructed based on this difference. Then, the initial network model is trained based on this target loss function to obtain the point cloud feature transformation network. In one possible implementation, the formula for calculating the target loss function can be as follows:

[0152] Loss l1 =||f_trans_all-f_point||

[0153] Among them, Loss l1 Let f_trans_all represent the target loss function, f_point represent the first predicted point cloud feature, and f_point represent the standard point cloud feature output by the point cloud encoder after training, based on the point cloud data reconstructed from the samples.

[0154] Based on the above Loss l1 By effectively optimizing f_trans_all, the point cloud feature transformation network learns the transformation relationship between image features and point cloud features, thereby training a point cloud feature transformation network with better performance, so as to achieve better reconstruction results in 3D reconstruction.

[0155] It should be noted that the image generation model, image encoder, point cloud feature transformation network, planar feature transformation network, and SDF network involved in the embodiments of this application can be trained together to improve the performance of each model in the 3D reconstruction process, thereby improving the reconstruction effect.

[0156] Based on the foregoing Figure 3 The corresponding embodiment describes the three-dimensional reconstruction process, and Figure 9 The training process of the point cloud feature transformation network described in the corresponding embodiments, and the overall technical framework diagram of the image processing method provided in this application embodiment can be found in [reference needed]. Figure 10 As shown, the entire technical solution includes two processes. The first process can be 3D reconstruction based on the point cloud data of the sample object. The second process can be to achieve image feature-point cloud feature alignment through a point cloud feature conversion network, that is, to convert image features into point cloud features.

[0157] The first process can be as follows: Figure 10 As shown in Figure 1001, in this process, point cloud data of the sample object is acquired, and then feature extraction is performed on the point cloud data through a point cloud encoder to obtain point cloud features (the point cloud features obtained in the first process can be second predicted point cloud features). Then, the second predicted point cloud features are transformed into three-plane features through a first initial network to obtain predicted three-plane features. Based on the predicted three-plane features, the sample object is reconstructed in three dimensions through the second initial network to obtain a predicted three-dimensional model. In order to utilize the difference between the predicted three-dimensional model and the standard three-dimensional model of the sample object, the point cloud encoder, the first initial network, and the second initial network are adjusted to train a three-dimensional generative network model. The three-dimensional generative network model includes a trained point cloud encoder, a planar feature transformation network trained based on the first initial network, and a generative network trained based on the second initial network. After obtaining the trained point cloud encoder, the point cloud features obtained by the trained point cloud encoder can be used as standard point cloud features, and the standard point cloud features can be used as supervision signals to train the point cloud feature transformation network.

[0158] The process of training a point cloud feature transformation network can be as follows: feature extraction is performed on the sample image of the sample object to obtain the sample image features. Then, the sample image features are transformed through an initial network model to obtain the first predicted point cloud features. Based on the difference between the first predicted point cloud features and the standard point cloud features, the initial network model is trained to obtain the point cloud feature transformation network.

[0159] The second process can be as follows: Figure 10 As shown in Figure 1002, in this process, only an initial image of the preset object needs to be input. Multiple images are generated through an image generation model, obtaining multiple images from different perspectives corresponding to the input image. For the generated multi-view images corresponding to the initial image, each multi-view image is a single image obtained by combining views corresponding to multiple perspectives. Then, two-dimensional images of the preset object from each perspective are cropped from the multi-view images. Next, feature extraction is performed on the two-dimensional images from each perspective using an image encoder to obtain image features. These image features are then converted into point cloud features using a trained point cloud feature transformation network (the point cloud features obtained in the second process can be point cloud features). The point cloud features are then subjected to a three-plane feature transformation to obtain three-plane features. Based on these three-plane features, the preset object is reconstructed in three dimensions using the SDF method to obtain a three-dimensional model of the preset object.

[0160] It should be noted that, based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods.

[0161] Based on the image processing method provided in the foregoing embodiments, this application also provides an image processing apparatus 1100. See also... Figure 11 As shown, the image processing device 1100 includes an acquisition unit 1101, an extraction unit 1102, a conversion unit 1103, and a generation unit 1104.

[0162] The acquisition unit 1101 is used to acquire a two-dimensional image of a preset object;

[0163] The extraction unit 1102 is used to extract features from the two-dimensional image to obtain the image features of the two-dimensional image;

[0164] The conversion unit 1103 is used to perform feature conversion on the image features through a point cloud feature conversion network to obtain corresponding point cloud features. The point cloud features are used to represent the three-dimensional spatial information of the preset object. The point cloud feature conversion network is trained by using standard point cloud features as supervision signals and sample images of sample objects as inputs. The trained point cloud feature conversion network has the function of converting image features into point cloud features. The point cloud features obtained by feature conversion have the same feature dimension as the standard point cloud features.

[0165] The generation unit 1104 is used to input the point cloud features into the model creation module, and the model creation module outputs a three-dimensional model of the preset object based on the point cloud features.

[0166] In one possible implementation, the acquisition unit 1101 is used for:

[0167] Obtain an initial image of the preset object, wherein the initial image is a single image of the preset object;

[0168] The initial image is generated using an image generation model to generate multi-view images of the preset object from multiple perspectives.

[0169] The two-dimensional images of the preset object corresponding to each of the multiple viewpoints are determined as the two-dimensional images of the preset object.

[0170] In one possible implementation, the acquisition unit 1101 is used for:

[0171] The initial image is generated by an image generation model to obtain a multi-view image corresponding to the initial image. The multi-view image is a single image obtained by combining the views corresponding to the multiple perspectives.

[0172] The preset object is cropped from the multi-view image to obtain a two-dimensional image of each of the multiple viewpoints.

[0173] In one possible implementation, the acquisition unit 1101 is used for:

[0174] Obtain control conditions, which are used to indicate the plurality of viewpoints;

[0175] According to the multiple perspectives indicated by the control conditions, the image generation model is controlled to generate two-dimensional images of the preset object corresponding to the multiple perspectives based on the initial image.

[0176] In one possible implementation, the extraction unit 1102 is used for:

[0177] Feature extraction is performed on the two-dimensional image from each viewpoint to obtain the image features of the two-dimensional image from each viewpoint.

[0178] The conversion unit 1103 is used for:

[0179] The point cloud feature transformation network is used to transform the image features of the two-dimensional image under each viewpoint to obtain the point cloud features of the two-dimensional image under each viewpoint.

[0180] The point cloud features of the two-dimensional image under each viewpoint are summed to obtain the corresponding point cloud features.

[0181] In one possible implementation, the model creation module includes a trained planar feature transformation network and a trained generative network, wherein the generative unit 1104 is used for:

[0182] The point cloud features are input into the planar feature transformation network, and the planar feature transformation network performs three-plane feature transformation on the point cloud features in the three-dimensional space formed by three orthogonal planes to obtain three-plane features. The three-plane features represent the voxel to which the point corresponding to the point cloud features belongs in the three-dimensional space.

[0183] The generative network outputs a 3D model of the preset object based on the three-plane features.

[0184] In one possible implementation, the generation unit 1104 is used for:

[0185] The first directed distance function value between the vertices of the voxels to which the three-plane features belong and the surface of the preset object is calculated through the generative network;

[0186] The generator network extracts vertices with a first directed distance function value of zero, and uses these vertices to generate the 3D model.

[0187] In one possible implementation, the apparatus further includes a training unit, the training unit being configured to:

[0188] Obtain the sample image of the sample object, and obtain the standard point cloud features;

[0189] Feature extraction is performed on the sample image to obtain the sample image features;

[0190] The sample image features are transformed using an initial network model to obtain the first predicted point cloud features.

[0191] Based on the difference between the first predicted point cloud features and the standard point cloud features, the initial network model is trained to obtain the point cloud feature transformation network.

[0192] In one possible implementation, the training unit is used for:

[0193] The point cloud features generated by the trained point cloud encoder are defined as the standard point cloud features. The point cloud features generated by the trained point cloud encoder are obtained by the trained point cloud encoder from the point cloud data of the sample object.

[0194] In one possible implementation, the training unit is used for:

[0195] Obtain the point cloud data of the sample object;

[0196] The point cloud encoder extracts features from the point cloud data to obtain the second predicted point cloud features.

[0197] The second predicted point cloud features are input into the initial creation module, and the initial creation module outputs a predicted 3D model of the sample object based on the second predicted point cloud features.

[0198] By utilizing the difference between the predicted 3D model and the standard 3D model of the sample object, the point cloud encoder and the initial creation module are adjusted to train a 3D generative network model, which includes the trained point cloud encoder and the model creation module trained based on the initial creation module.

[0199] In one possible implementation, the initial creation module includes a first initial network and a second initial network, and the training unit is used for:

[0200] The first initial network performs a three-plane feature transformation on the second predicted point cloud features in the three-dimensional space formed by three orthogonal planes to obtain predicted three-plane features. The predicted three-plane features represent the voxel to which the point corresponding to the second predicted point cloud features belongs in the three-dimensional space.

[0201] The predicted three-plane features are input into the second initial network, and the second initial network outputs the predicted three-dimensional model based on the predicted three-plane features.

[0202] By utilizing the difference between the predicted 3D model and the standard 3D model of the sample object, the point cloud encoder, the first initial network, and the second initial network are adjusted to train the 3D generative network model, which includes the trained point cloud encoder, the planar feature transformation network trained based on the first initial network, and the generative network trained based on the second initial network.

[0203] In one possible implementation, the training unit is used for:

[0204] The second initial network is used to calculate the second directed distance function value between the vertex of the voxel to which the predicted three-plane feature belongs and the surface of the sample object;

[0205] The vertices with a second directed distance function value of zero are extracted through the second initial network, and the predicted 3D model is generated using the vertices with a second directed distance function value of zero.

[0206] As can be seen from the above technical solutions, when it is necessary to perform 3D reconstruction of a preset object, a 2D image of the preset object can be obtained, and feature extraction can be performed on the 2D image to obtain the image features of the 2D image. However, this application does not directly perform 3D reconstruction based on image features, but instead uses a point cloud feature conversion network to perform feature conversion on the image features to obtain the corresponding point cloud features. The point cloud feature conversion network is trained using standard point cloud features, i.e., real point cloud features, as the supervision signal and sample images of the sample object as input. During the training process, the point cloud feature conversion network learns the conversion relationship between image features and point cloud features, enabling the trained point cloud feature conversion network to have the function of converting image features into point cloud features. Therefore, point cloud features can be accurately obtained based on image features through the point cloud feature conversion network. Point cloud features can represent the complete three-dimensional spatial information of a preset object and reflect the three-dimensional geometry of the object's surface. They can be used to understand and reconstruct the object from multiple angles and levels, providing richer content for 3D reconstruction. Thus, when the model creation module outputs a 3D model of a preset object based on point cloud features, even if some parts are not visible, the information of these invisible areas can be accurately inferred and supplemented through point cloud features from other angles, thereby improving the reconstruction effect of the 3D model and avoiding serious geometric deformation, singularity, or geometric blurring.

[0207] This application also provides a computer device capable of executing an image processing method. The computer device may be a terminal. Figure 12 This diagram illustrates the structure of a terminal according to an embodiment of this application. Figure 12 In this example, taking a smartphone as the terminal:

[0208] refer to Figure 12 A smartphone includes components such as a radio frequency (RF) circuit 1210, a memory 1220, an input unit 1230, a display unit 1240, a sensor 1250, an audio circuit 1260, a Wi-Fi module 1270, a processor 1280, and a power supply 1290. The input unit 1230 may include a touch panel 1231 and other input devices 1232, the display unit 1240 may include a display panel 1241, and the audio circuit 1260 may include a speaker 1261 and a microphone 1262. It is understood that... Figure 12The smartphone structure shown does not constitute a limitation on smartphones and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0209] The memory 1220 can be used to store software programs and modules. The processor 1280 executes various functions and data processing of the smartphone by running the software programs and modules stored in the memory 1220. The memory 1220 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, applications required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the smartphone (such as audio data, phonebook, etc.). In addition, the memory 1220 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0210] The processor 1280 is the control center of the smartphone, connecting various parts of the smartphone via various interfaces and lines. It performs various functions and processes data by running or executing software programs and / or modules stored in the memory 1220 and calling data stored in the memory 1220. Optionally, the processor 1280 may include one or more processing units; preferably, the processor 1280 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 1280.

[0211] In this embodiment, the processor 1280 in the smartphone can execute the image processing methods provided in the various embodiments of this application.

[0212] The computer device provided in this application embodiment can also be a server. Please refer to [link / reference]. Figure 13 As shown, Figure 13This is a structural diagram of a server 1300 provided in an embodiment of this application. The server 1300 can vary significantly due to different configurations or performance. It may include one or more processors, such as a Central Processing Unit (CPU) 1322, and a memory 1332, and one or more storage media 1330 (e.g., one or more mass storage devices) for storing application programs 1342 or data 1344. The memory 1332 and storage media 1330 can be temporary or persistent storage. The program stored in the storage media 1330 may include one or more modules (not shown in the diagram), each module including a series of instruction operations on the server. Furthermore, the CPU 1322 may be configured to communicate with the storage media 1330 and execute the series of instruction operations stored in the storage media 1330 on the server 1300.

[0213] Server 1300 may also include one or more power supplies 1326, one or more wired or wireless network interfaces 1350, one or more input / output interfaces 1358, and / or one or more operating systems 1341, such as Windows Server. TM Mac OS X TM Unix TM Linux TM FreeBSD TM etc.

[0214] In this embodiment, the central processing unit 1322 in the server 1300 can execute the image processing methods provided in the various embodiments of this application.

[0215] According to one aspect of this application, a computer-readable storage medium is provided for storing a computer program for performing the image processing methods described in the foregoing embodiments.

[0216] According to one aspect of this application, a computer program product is provided, comprising a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the methods provided in various optional implementations of the above embodiments.

[0217] The descriptions of the processes or structures corresponding to the above figures each have their own emphasis. For parts of a process or structure that are not described in detail, please refer to the relevant descriptions of other processes or structures.

[0218] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0219] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0220] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0221] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0222] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a terminal, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing computer programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0223] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0224] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. An image processing method, characterized by, The method comprises: acquiring a two-dimensional image of a preset object; extracting features from the two-dimensional image to obtain image features of the two-dimensional image; performing feature conversion on the image features through a point cloud feature conversion network to obtain corresponding point cloud features, the point cloud features being used to represent three-dimensional spatial information of the preset object, the point cloud feature conversion network being obtained by training with a standard point cloud feature as a supervision signal and a sample image of a sample object as input, the point cloud feature conversion network having a function of converting image features into point cloud features, the point cloud features obtained through feature conversion having the same feature dimension as the standard point cloud features; inputting the point cloud features into a model creation module, and outputting a three-dimensional model of the preset object based on the point cloud features through the model creation module.

2. The method of claim 1, wherein, The acquiring of the two-dimensional image of the preset object comprises: acquiring an initial image of the preset object, the initial image being a single image of the preset object; generating multiple-view images from the initial image through an image generation model to obtain two-dimensional images corresponding to the preset object under multiple views respectively; determining the two-dimensional images corresponding to the preset object under multiple views respectively as the two-dimensional images of the preset object.

3. The method of claim 2, wherein, The generating of the multiple-view images from the initial image through the image generation model to obtain the two-dimensional images corresponding to the preset object under multiple views respectively comprises: generating multiple-view images from the initial image through an image generation model to obtain the multiple-view images corresponding to the initial image, the multiple-view images being single images obtained by combining views corresponding to the multiple views respectively; cropping the two-dimensional images of the preset object under each of the multiple views from the multiple-view images.

4. The method of claim 2, wherein, The generating of the multiple-view images from the initial image through the image generation model to obtain the two-dimensional images corresponding to the preset object under multiple views respectively comprises: acquiring a control condition, the control condition being used to indicate the multiple views; controlling the image generation model to generate the two-dimensional images corresponding to the preset object under the multiple views respectively based on the initial image according to the multiple views indicated by the control condition.

5. The method of claim 2, wherein, The extracting of features from the two-dimensional image to obtain image features of the two-dimensional image comprises: extracting features from the two-dimensional images under each view respectively to obtain image features of the two-dimensional images under each view; The performing of feature conversion on the image features through the point cloud feature conversion network to obtain corresponding point cloud features comprises: performing feature conversion on the image features of the two-dimensional images under each view through the point cloud feature conversion network to obtain point cloud features of the two-dimensional images under each view; summing the point cloud features of the two-dimensional images under each view to obtain corresponding point cloud features.

6. The method of claim 1, wherein, The model creation module comprises a plane feature conversion network and a generation network that have completed training, and the inputting of the point cloud features into the model creation module and the outputting of the three-dimensional model of the preset object based on the point cloud features through the model creation module comprise: inputting the point cloud feature into the plane feature conversion network, performing three-plane feature conversion on the point cloud feature in a three-dimensional space formed by three orthogonal planes through the plane feature conversion network, to obtain three-plane features, the three-plane features representing voxels to which points corresponding to the point cloud feature belong in the three-dimensional space; outputting, through the generation network, a three-dimensional model of the preset object based on the three-plane features.

7. The method of claim 6, wherein, The outputting, through the generation network, a three-dimensional model of the preset object based on the three-plane features includes: calculating, through the generation network, a first directed distance function value between a vertex of a voxel to which the three-plane features belong and a surface of the preset object; extracting, through the generation network, a vertex at which the first directed distance function value is zero, and generating the three-dimensional model using the vertex at which the first directed distance function value is zero.

8. The method according to any one of claims 1 to 7, characterized in that, The method further includes: obtaining a sample image of the sample object, and obtaining the standard point cloud feature; performing feature extraction on the sample image to obtain a sample image feature of the sample image; performing feature conversion on the sample image feature through an initial network model to obtain a first predicted point cloud feature; training the initial network model based on a difference between the first predicted point cloud feature and the standard point cloud feature, to obtain the point cloud feature conversion network.

9. The method of claim 8, wherein, The obtaining the standard point cloud feature includes: determining a point cloud feature generated by a trained point cloud encoder as the standard point cloud feature, the point cloud feature generated by the trained point cloud encoder being obtained by performing feature extraction on point cloud data of the sample object by the trained point cloud encoder.

10. The method of claim 9, wherein, The method further includes: obtaining point cloud data of the sample object; performing feature extraction on the point cloud data through the point cloud encoder to obtain a second predicted point cloud feature; inputting the second predicted point cloud feature into an initial creation module, and outputting, through the initial creation module, a predicted three-dimensional model of the sample object based on the second predicted point cloud feature; adjusting the point cloud encoder and the initial creation module based on a difference between the predicted three-dimensional model and a standard three-dimensional model of the sample object, to obtain a three-dimensional generation network model, the three-dimensional generation network model including the trained point cloud encoder and a model creation module trained based on the initial creation module.

11. The method of claim 10, wherein, The initial creation module includes a first initial network and a second initial network, and the inputting the second predicted point cloud feature into the initial creation module and outputting, through the initial creation module, a predicted three-dimensional model of the sample object based on the second predicted point cloud feature includes: performing three-plane feature conversion on the second predicted point cloud feature in a three-dimensional space formed by three orthogonal planes through the first initial network, to obtain predicted three-plane features, the predicted three-plane features representing voxels to which points corresponding to the second predicted point cloud feature belong in the three-dimensional space; inputting the predicted three-plane features into the second initial network, and outputting, through the second initial network, the predicted three-dimensional model based on the predicted three-plane features; The difference between the predicted three-dimensional model and the standard three-dimensional model of the sample object is used to adjust the point cloud encoder and the initial creation module, and a three-dimensional generation network model is trained, including: The difference between the predicted three-dimensional model and the standard three-dimensional model of the sample object is used to adjust the point cloud encoder, the first initial network and the second initial network, and the three-dimensional generation network model is trained, including the trained point cloud encoder, the plane feature conversion network trained based on the first initial network and the generation network trained based on the second initial network.

12. The method of claim 11, wherein, The second initial network is used to output the predicted three-dimensional model based on the predicted three-plane features. The second initial network is used to calculate a second directed distance function value between the vertices of the voxels to which the predicted three-plane features belong and the surface of the sample object. The second initial network is used to extract the vertices with a zero second directed distance function value, and the predicted three-dimensional model is generated based on the vertices with a zero second directed distance function value.

13. An image processing apparatus characterized by comprising: The device includes an acquisition unit, an extraction unit, a conversion unit and a generation unit: The acquisition unit is configured to acquire a two-dimensional image of a preset object. The extraction unit is configured to perform feature extraction on the two-dimensional image to obtain image features of the two-dimensional image. The conversion unit is configured to perform feature conversion on the image features by a point cloud feature conversion network to obtain corresponding point cloud features, the point cloud features being used to represent three-dimensional space information of the preset object, the point cloud feature conversion network being trained based on a standard point cloud feature as a supervision signal and a sample image of a sample object as an input, the trained point cloud feature conversion network having a function of converting image features into point cloud features, and the point cloud features obtained by feature conversion having the same feature dimension as the standard point cloud features. The generation unit is configured to input the point cloud features into a model creation module, and output a three-dimensional model of the preset object based on the point cloud features by the model creation module.

14. A computer device, comprising: The computer device includes a processor and a memory: The memory is configured to store a computer program and transmit the computer program to the processor. The processor is configured to execute the method according to the instructions in the computer program.

15. A computer-readable storage medium, characterized in that, The computer readable storage medium is configured to store a computer program, and the computer program causes the processor to execute the method according to any one of claims 1-12 when executed by the processor. The computer readable storage medium is configured to store a computer program, and the computer program causes the processor to execute the method according to any one of claims 1-12 when executed by the processor.

Images