Model training and geometric model generation method, equipment, medium and product

Abstract

The embodiment of the invention provides a model training and geometric model generation method, equipment, a medium and a product. Wherein the training process is to obtain a first sample object geometric model and a sample reference image containing a second sample object and corresponding to a preset view angle, and execute the following training steps for a plurality of to-be-trained view angles determined by the preset view angle, that is, for the current to-be-trained view angle, the to-be-trained view angle is obtained; rendering the first sample object geometric model to generate a sample original image corresponding to the current visual angle to be trained; based on the sample reference image and the sample original image, an image generation model is trained for the current to-be-trained view angle until a sample reconstruction image generated by the image generation model when the training requirement is met is obtained; and after the first sample object geometric model is updated based on the sample reconstruction image meeting the training requirement, the training step is executed for the next to-be-trained view angle. When the image generation model trained in the embodiment of the invention constructs the target object geometric model, the accuracy of the target object geometric model can be improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence technology, and in particular to a method, device, medium, and product for model training and geometric model generation. Background Art

[0002] With the development of computer technology, the construction of geometric models of objects such as humans and animals, such as 3D Gaussian models, has become increasingly popular across various industries. Geometric models represent the geometry and appearance of objects, and can be used to generate realistic visual effects through rendering technology. For example, a 3D Gaussian model of a human face can be used to render a digital human, such as a virtual anchor.

[0003] Geometric models usually require manual modeling to implement, but in actual applications, there is a need to change the object image of the geometric model. For example, in digital human application scenarios, it is hoped to render a digital human with a customized face. In order to avoid manual modeling operations, existing solutions can usually use artificial intelligence models to render and generate reconstructed images from different perspectives based on the pre-built original geometric model of the original object and the customized target object image, and then map the reconstructed images from different perspectives to the original geometric model to obtain the target geometric model corresponding to the target object. However, the processing results of the artificial intelligence model are not accurate, resulting in the problem of low accuracy of the constructed target geometric model, which urgently needs to be improved. Summary of the Invention

[0004] The embodiments of the present application provide a model training and geometric model generation method, device, medium and product to solve the problem of low accuracy of processing results in the prior art.

[0005] In a first aspect, an embodiment of the present application provides a model training method for training an image generation model, comprising:

[0006] Acquire a geometric model of a first sample object and a sample reference image containing a second sample object corresponding to a preset viewing angle;

[0007] Based on the preset perspective, determining a plurality of perspectives to be trained;

[0008] For the multiple perspectives to be trained, a training step is performed, the training step comprising:

[0009] For the current viewing angle to be trained, rendering the first sample object geometric model to generate a sample original image corresponding to the current viewing angle to be trained;

[0010] Based on the sample reference image and the sample original image, using the image generation model, generate a sample reconstructed image corresponding to the current viewing angle to be trained;

[0011] The image generation model is trained based on difference information of the sample reconstructed image and the sample reference image until a sample reconstructed image meeting a training requirement is obtained by using the image generation model;

[0012] The sample reconstructed image meeting the training requirement is mapped to the first sample object geometric model, the first sample object geometric model is updated, and the training step is performed for a next to-be-trained view angle.

[0013] In a second aspect, an embodiment of the present application provides a geometric model generation method, comprising:

[0014] A plurality of target original images containing a first object corresponding to a plurality of view angles and a target reference image containing a second object corresponding to a preset view angle are obtained, and the plurality of view angles are obtained based on adjustment of the preset view angle;

[0015] Based on the plurality of target original images and the target reference image, an image generation model is used to generate target reconstructed images corresponding to the plurality of view angles respectively;

[0016] A target object geometric model is generated based on the target reconstructed images corresponding to the plurality of view angles respectively;

[0017] The image generation model is trained by using a plurality of sample original images containing a first sample object corresponding to a plurality of to-be-trained view angles and a sample reference image containing a second sample object corresponding to the preset view angle; a sample reconstructed image meeting a training requirement obtained under any to-be-trained view angle is used to map to the first sample object geometric model to update the first sample object geometric model; and a sample original image corresponding to any to-be-trained view angle is obtained based on the first sample object geometric model after the update.

[0018] In a third aspect, an embodiment of the present application provides a computing device, comprising a processing component and a storage component; the storage component stores one or more computer instructions; the one or more computer instructions are used to be called and executed by the processing component to implement the model training method of the first aspect or the geometric model generation method of the second aspect.

[0019] In a fourth aspect, an embodiment of the present application provides a computer storage medium storing a computer program, and the computer program is executed by a computer to implement the model training method of the first aspect or the geometric model generation method of the second aspect.

[0020] In a fifth aspect, an embodiment of the present application provides a computer program product, comprising a computer program or instructions, which, when executed by a computer, implements the model training method of the first aspect described above, or implements the geometric model generation method of the second aspect described above.

[0021] In an embodiment of the present application, when training an image generation model, a first sample object geometric model and a sample reference image containing a second sample object corresponding to a preset perspective are obtained. Based on the preset perspective, multiple perspectives to be trained are determined. For each of the multiple perspectives to be trained, the following training steps are performed: For the current perspective to be trained, the first sample object geometric model is rendered into a sample original image corresponding to the current perspective to be trained; based on the sample reference image and the sample original image, a sample reconstructed image corresponding to the current perspective to be trained is generated using the image generation model; and based on the difference information between the sample reconstructed image and the sample reference image, the image generation model is trained until a sample reconstructed image that meets the training requirements is obtained using the image generation model; and then, based on the sample reconstructed image that meets the training requirements, the first sample object geometric model is updated. The updated first sample object geometric model is used to participate in the training step corresponding to the next perspective to be trained. The solution of the embodiment of the present application sequentially trains the image generation model for multiple perspectives (i.e., multiple perspectives to be trained). Compared with training the model for multiple perspectives simultaneously, this helps the image generation model better learn the algorithm for image reconstruction at each perspective, thereby improving the accuracy of the reconstructed image generated for each perspective. In addition, after the training for any perspective is completed, the embodiment of the present application will also update the first sample object geometric model based on the sample reconstructed image generated when the training requirements are met, and render the sample original image for the next perspective training based on the updated first sample object geometric model. This helps the image generation model to learn the spatial correspondence between the perspective and the reconstructed images of other perspectives during the training for any perspective, thereby improving the spatial consistency of the reconstructed images generated from different perspectives, and further ensuring that the image generation model trained using the embodiment of the present application, based on the target reference image and the target original images corresponding to multiple perspectives, respectively, generates target reconstructed images corresponding to multiple perspectives with spatial consistency at different perspectives, that is, improves the accuracy of the image generation model processing results, and further improves the accuracy of the target object geometric model constructed based on the target reconstructed image at multiple perspectives.

[0022] These and other aspects of the present application will become more readily apparent from the description of the following embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0024] Figure 1 A flowchart of an embodiment of the model training method provided by the present application is shown;

[0025] Figure 2 shows a schematic structural diagram of the image generation model provided by this application;

[0026] Figure 3 A flowchart of another embodiment of the model training method provided by the present application is shown;

[0027] Figure 4 A flow chart of an embodiment of a method for generating a geometric model provided by the present application is shown;

[0028] Figure 5 A schematic diagram of a process for training an image generation model in a practical application provided by the present application is shown;

[0029] Figure 6 A schematic diagram of a process for generating a geometric model of a target object in a practical application provided by the present application is shown;

[0030] Figure 7 A schematic structural diagram of an embodiment of a model training device provided by the present application is shown;

[0031] Figure 8 A schematic structural diagram of an embodiment of a geometric model generating device provided by the present application is shown;

[0032] Figure 9 A schematic structural diagram of an embodiment of a computing device provided by the present application is shown. DETAILED DESCRIPTION

[0033] To make the purpose, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the specific embodiments of this application and the corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0034] It should be noted that, in the case of user information involved in the embodiments of this application, the user information (including but not limited to user device information, user personal information, user images, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse. In addition, the various models involved in this application (including but not limited to language models or large models) are in compliance with relevant laws and standards.

[0035] With the development of computer technology, the construction of geometric models of objects such as humans and animals, such as 3D Gaussian models, has become increasingly popular across various industries. Geometric models represent the geometry and appearance of objects, and can be used to generate realistic visual effects through rendering technology. For example, a 3D Gaussian model of a human face can be used to render a digital human to play the role of a virtual anchor.

[0036] In order to meet the demand for changing the object image of the geometric model in practical applications, existing solutions usually use artificial intelligence models to edit the features of certain characteristic parts in the existing object geometric model into the features of the target object, thereby obtaining the target object geometric model. For example, an image generation model can be used to render and generate reconstructed images of different perspectives based on the pre-built original geometric model of the original object and a customized target object image, and then the reconstructed images of different perspectives are mapped to the original geometric model to obtain the target geometric model corresponding to the target object. The target geometric model can then be rendered to generate an image or video with the image of the target object. However, the target geometric model constructed in this way is not effective and has the problem of low accuracy. The inventors have found that the target geometric model constructed by the reconstructed images of different perspectives generated by the image generation model often has spatial inconsistencies. For example, the reconstructed image of a certain perspective cannot be aligned with the reconstructed image of another perspective, resulting in a poor target geometric model. Therefore, the final constructed target geometric model is not effective, which is directly related to the quality of the reconstructed image generated by the image generation model. The inventors also found that the image generation model of the existing scheme often uses a pre-trained model, such as a language model or a multimodal model, without fine-tuning training. However, if only the reconstructed image of a certain perspective is used as a training label for model training, the final result is still not ideal.

[0037] In response to the above-mentioned problems, the inventors have proposed the technical solution of the present application after a series of studies. The basic idea is: when training the image generation model, a first sample object geometric model and a sample reference image containing a second sample object corresponding to a preset perspective are obtained; based on the preset perspective, multiple perspectives to be trained are determined; for the multiple perspectives to be trained, the following training steps are performed, namely, for the current perspective to be trained, the first sample object geometric model is rendered to generate a sample original image corresponding to the current perspective to be trained; based on the sample reference image and the sample original image, the image generation model is used to generate a sample reconstructed image corresponding to the current perspective to be trained, and based on the difference information between the sample reconstructed image and the sample reference image, the image generation model is trained until a sample reconstructed image that meets the training requirements is obtained using the image generation model; and then the first sample object geometric model is updated based on the sample reconstructed image that meets the training requirements, and the updated first sample object geometric model is used to participate in the training step corresponding to the next perspective to be trained. The solution of the embodiment of the present application sequentially trains the image generation model for multiple perspectives (i.e., multiple perspectives to be trained). Compared with training the model for multiple perspectives at the same time, it helps the image generation model better learn the algorithm for image reconstruction at each perspective, thereby improving the accuracy of the reconstructed image generated for each perspective. In addition, after the embodiment of the present application completes the training for any perspective, it will also update the first sample object geometric model based on the sample reconstructed image generated when the training requirements are met, and render the sample original image for the next perspective training based on the updated first sample object geometric model, which helps the image generation model learn the spatial correspondence between the reconstructed image of the perspective and other perspectives during the training process for any perspective, thereby improving the spatial consistency of the reconstructed images generated from different perspectives, thereby ensuring that the image generation model trained using the embodiment of the present application, based on the target reference image and the target original images corresponding to the multiple perspectives, has spatial consistency at different perspectives, that is, improves the accuracy of the image generation model processing results, thereby improving the accuracy of the target object geometric model constructed based on the target reconstructed image at multiple perspectives.

[0038] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without making creative efforts are within the scope of protection of this application.

[0039] The implementation details of the technical solution of the embodiment of the present application are described in detail below.

[0040] Figure 1This is a flowchart of an embodiment of the model training method provided in this application, which is applied to the training of an image generation model. The technical solution of this embodiment can be executed by a processing end, which can be a server in an online system, or of course, other nodes independent of the server in the online system. The processing end or server can include servers that provide various services, such as servers for background training that support models used on the user end.

[0041] It should be noted that the processing end or service end can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. The server can also be a server in a distributed system, or a server integrated with blockchain. The server can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0042] Figure 1 The model training method shown can include the following steps:

[0043] S101 , obtaining a geometric model of a first sample object and a sample reference image containing a second sample object corresponding to a preset viewing angle.

[0044] Among them, the sample object of this embodiment can be a specific part image of a physical entity (such as a person and an animal, etc.) or a virtual object (such as an animated character). For example, the sample object of this embodiment can be an image of the upper body or the whole body of a person, etc. The first sample object and the second sample object of this embodiment are of the same object type, but have different object features. For example, if the sample object is an image of the upper body of a person, then the first sample object and the second sample object at this time can be the upper body images corresponding to two people with different looks. It should be noted that in the actual model training process, the number of the first sample object and the second sample object is usually multiple. This embodiment only provides an introduction to the method of how to perform model training on the image generation model for any first sample object and the second sample object.

[0045] The first sample object geometric model can be a digital image constructed using specialized software and digital modeling techniques in computer graphics, used to accurately represent the morphology, texture, and dynamic characteristics of a specific part of a physical entity or fictional object. Optionally, the first sample object geometric model in this embodiment can be a 3D Gaussian model generated by high-quality 3D rendering using 3D Gaussian sputtering technology, which describes the sample object scene using a 3D Gaussian distribution and utilizes this Gaussian distribution for efficient rasterization processing.

[0046] Since the geometric model is essentially a precise reproduction of the imaging effect of the camera shooting angle through digital modeling and optical simulation, the so-called angle of view in this embodiment can be the camera shooting angle of view. Among them, the selection of the preset angle of view can be an angle of view that can fully and clearly display the features of the object. For example, it can be the positive angle of view of the camera. The sample reference image can be a two-dimensional image containing the second sample object that is photographed or drawn at a preset angle of view. In some embodiments, if the image generation model is used to transfer part of the characteristic parts of the second sample object to the geometric model of the first sample object, then the sample reference image at this time must at least contain the part of the characteristic parts. For example, if the second sample object is the upper body image of a person, and the image generation model is used to transfer the facial features in the second sample object to the geometric model of the first sample object, then the sample reference image at this time can be a clear facial image corresponding to the second sample object.

[0047] Optionally, this embodiment can select two sample objects of the same type but with different object features as the first sample object and the second sample object based on the object type corresponding to the image to be reconstructed in the actual reasoning stage of the image generation model. Then, for the first sample object, if its corresponding geometric model has been constructed in advance, the geometric model can be directly obtained as the first sample object geometric model. If it has not been constructed in advance, the first sample object geometric model can be constructed based on the image containing the first sample object under multiple perspectives. For example, a three-dimensional Gaussian model of the first sample object can be constructed based on the three-dimensional Gaussian sputtering technology as the first sample object geometric model. When obtaining the sample reference image, an image containing the second sample object under a preset perspective can be taken or drawn as the sample reference image.

[0048] S102: Determine a plurality of viewing angles to be trained based on preset viewing angles.

[0049] Since the target object geometric model presents a large number of frontal or near-frontal perspectives when applied in various fields, this embodiment of the present application can select a perspective close to the preset perspective when determining multiple perspectives to be trained, that is, the number of frontal perspectives is greater than the number of side perspectives, and the number of side perspectives is greater than the number of back perspectives. The selection of the specific range and number of multiple perspectives to be trained can be determined through a large number of experimental statistics. Multiple perspectives to be trained can be determined based on the preset perspective. Specifically, the multiple perspectives to be trained can be selected within a range of 90 degrees to the left and right of the preset perspective (such as the front perspective). For example, the multiple perspectives to be trained can be evenly divided within the perspective range after the preset perspective is expanded, or the multiple perspectives to be trained can be randomly divided, or the multiple perspectives to be trained can be divided according to a method of differentiating multiple values ​​from the preset perspective (the multiple values ​​are arranged from small to large).

[0050] It should be noted that the object geometry model can reproduce imaging effects from multiple perspectives. The multiple perspectives referred to in this and subsequent embodiments are multiple perspectives to be trained. This means that during the model training phase, the image generation model needs to be trained sequentially for each of the multiple perspectives. Therefore, these multiple perspectives are also referred to as multiple perspectives to be trained.

[0051] For multiple viewing angles to be trained, a training step is performed, which includes:

[0052] S103 : Rendering the first sample object geometric model for the current viewing angle to be trained to generate a sample original image corresponding to the current viewing angle to be trained.

[0053] In order to ensure the accuracy of the target object geometric model subsequently constructed based on the image generation model under multiple viewpoints, this embodiment requires training the image generation model for multiple viewpoints (i.e., multiple to-be-trained viewpoints). Specifically, each of the multiple to-be-trained viewpoints is sequentially used as the current to-be-trained viewpoint to train the image generation model. The first selected current to-be-trained viewpoint can be a set viewpoint from the multiple to-be-trained viewpoints, or it can be a viewpoint randomly selected from the multiple to-be-trained viewpoints. Each subsequent current to-be-trained viewpoint selected can be related to the previous current to-be-trained viewpoint (e.g., selecting the viewpoint closest to the previous current to-be-trained viewpoint), or it can be a randomly selected, untrained viewpoint.

[0054] Optionally, when training the image generation model based on the current perspective to be trained, this embodiment can render the first sample object geometric model at the current perspective to be trained, and obtain a rendered image corresponding to the current perspective to be trained as the sample original image corresponding to the current perspective to be trained. For example, the observation perspective corresponding to the current perspective to be trained can be first determined based on camera parameters (position, direction, projection mode), and then the vertex coordinates of the first sample object geometric model are converted to the camera coordinate system based on the observation perspective and perspective projection is performed. Then, the perspective-projected geometric body is converted into a two-dimensional pixel grid through rasterization. Finally, the pixel color is calculated based on the lighting, material, and texture information, and post-processing optimization is performed to obtain the sample original image at the current perspective to be trained.

[0055] S104: Based on the sample reference image and the sample original image, an image generation model is used to generate a sample reconstructed image corresponding to the current viewing angle to be trained.

[0056] Optionally, this embodiment may be to input the sample reference image and the sample original image into an image generation model, which will extract features from the sample reference image and the sample original image respectively, and perform image reconstruction based on the extracted features, so as to replace certain or all feature parts in the sample original image with corresponding feature parts in the sample reference image. For example, if the sample original image is an image of the upper body geometric model of person A rendered at the current training perspective, and the sample reference image is a frontal upper body image of person B, then based on the features extracted from the sample original image and the sample reference image, all upper body features of the upper body geometric model of person A are replaced with the upper body features of person B; alternatively, only the facial features of the upper body geometric model of person A are replaced with the facial features of person B, while other parts retain the features of person A.

[0057] It should be noted that the image generation model further includes a feature extraction network and an image generation network. For example, the feature extraction network extracts features from the sample reference image and the sample original image; the image generation network reconstructs the image based on the extracted features. The process of how these networks cooperate to generate the sample reconstructed image will be detailed in subsequent examples.

[0058] S105 , training the image generation model based on the difference information between the sample reconstructed image and the sample reference image, until a sample reconstructed image that meets the training requirements is obtained by using the image generation model.

[0059] Optionally, the purpose of training the image generation model in this embodiment is to improve the accuracy of its reconstructed images at multiple viewing angles, that is, to reduce the feature distance between the sample reconstructed image and the sample reference image at multiple viewing angles, so that the object features of the sample reconstructed image are as close as possible to the sample reference image at any viewing angle. Therefore, this embodiment can determine a model training loss based on the difference information between the sample reconstructed image and the sample reference image, and adjust the model parameters of the image generation model based on the model training loss. In this embodiment, the above training steps (i.e., S103-S104) can be repeated for the current viewing angle to be trained, using sample original images rendered at the current viewing angle to be trained based on different first sample object geometric models, and sample reference images containing different second sample objects. The image generation model is trained for the current viewing angle to be trained until the image generation model meets the training conditions at the current viewing angle to be trained. For example, the image generation model has been trained a preset number of times for the current viewing angle to be trained. Alternatively, the difference between the sample reconstructed image and the sample reference image for the current viewing angle to be trained is sufficiently small, such as when the model training loss is less than a loss threshold. In this embodiment, when the image generation model has met the training requirements for the current viewing angle to be trained, the model training process for the current viewing angle to be trained is stopped, and a sample reconstructed image generated by the image generation model when the training conditions are met is obtained.

[0060] S106 , mapping the sample reconstructed image that meets the training requirements to the first sample object geometric model, updating the first sample object geometric model, and performing a training step for the next viewing angle to be trained.

[0061] Optionally, after obtaining a sample reconstructed image that meets the training requirements, this embodiment needs to map the sample reconstructed image that meets the training requirements to a first sample object geometric model to achieve an update of the first sample object geometric model. The specific mapping process can be to update the first sample object geometric model based on the obtained sample reconstructed image. For example, it can be to extract geometric and semantic information from the sample reconstructed image, estimate the object pose and position in combination with the camera parameters corresponding to the current training perspective, spatially align the extracted data with the first sample object geometric model, and finally adjust the model vertices, textures, or topological structures through an optimization algorithm to achieve the update of the first sample object geometric model. The updated first sample object geometric model is used to participate in the above-mentioned training steps corresponding to the next training perspective.

[0062] Specifically, the next untrained view among the multiple to-be-trained viewpoints is used as the new current to-be-trained viewpoint (i.e., the current to-be-trained viewpoint corresponding to the next iterative training). For example, a viewpoint may be randomly selected from the untrained viewpoints as the new current to-be-trained viewpoint, or the untrained viewpoint closest to the current to-be-trained viewpoint may be selected as the new current to-be-trained viewpoint. Based on the updated first sample object geometric model and the new current to-be-trained viewpoint, steps S103-S106 are executed again until each of the multiple to-be-trained viewpoints is used as the current to-be-trained viewpoint, at which point the training step for the image generation model is terminated.

[0063] It should be noted that the image generation model trained in this embodiment can be used to generate target reconstructed images corresponding to multiple perspectives based on target original images containing a first object, each corresponding to a plurality of perspectives, and a target reference image containing a second object, corresponding to a preset perspective. The obtained target reconstructed images corresponding to the plurality of perspectives can be used to generate a target object geometric model corresponding to the second target object. The specific implementation method will be described in detail in subsequent embodiments and will not be elaborated here.

[0064] In addition, the image generation model involved in this article can be a machine learning model, such as a neural network model, and specifically can be a content generation model based on generative artificial intelligence. In addition, the embodiment of the present application does not limit the number of model parameters supported by the model, with the goal of meeting actual needs.

[0065] The technical solution of this embodiment obtains a first sample object geometric model and a sample reference image containing a second sample object corresponding to a preset perspective. Based on the preset perspective, multiple perspectives to be trained are determined. For the multiple perspectives to be trained, the following training steps are performed: for the current perspective to be trained, the first sample object geometric model is rendered to generate a sample original image corresponding to the current perspective to be trained; based on the sample reference image and the sample original image, an image generation model is used to generate a sample reconstructed image corresponding to the current perspective to be trained; and based on the difference information between the sample reconstructed image and the sample reference image, the image generation model is trained until a sample reconstructed image that meets the training requirements is obtained using the image generation model; and then, based on the sample reconstructed image that meets the training requirements, the first sample object geometric model is updated. The updated first sample object geometric model is used to participate in the training step corresponding to the next perspective to be trained. The solution of the embodiment of the application sequentially trains the image generation model for multiple perspectives. Compared with training the model for multiple perspectives simultaneously, it helps the image generation model better learn the algorithm for image reconstruction at each perspective, thereby improving the accuracy of the reconstructed image generated for each perspective. In addition, after the training for any perspective is completed, the embodiment of the present application will also update the first sample object geometric model based on the sample reconstructed image generated when the training requirements are met, and render the sample original image for the next perspective training based on the updated first sample object geometric model. This helps the image generation model to learn the spatial correspondence between the perspective and the reconstructed images of other perspectives during the training for any perspective, thereby improving the spatial consistency of the reconstructed images generated from different perspectives, and further ensuring that the image generation model trained using the embodiment of the present application is based on the target reference image and the target original images corresponding to multiple perspectives, and the target reconstructed images corresponding to multiple perspectives generated have spatial consistency at different perspectives, that is, the accuracy of the image generation model processing results is improved, and further the accuracy of the target object geometric model constructed based on the target reconstructed image at multiple perspectives is improved.

[0066] In some embodiments, in order to further improve the accuracy of reconstructed image generation, this embodiment may be to perform image segmentation processing on the sample original image to obtain a sample segmentation image corresponding to the first characteristic part; based on the sample reference image, the sample original image, and the sample segmentation image, use the image generation model to generate a sample reconstructed image corresponding to the current viewing angle to be trained. Among them, the first characteristic part can be other characteristic parts in the sample original image except the part that needs to be feature migrated (i.e., the second characteristic part). For example, if the sample original image is rendered based on the upper body geometric model of a person, and the part that needs to be feature migrated is the person's face, then the first characteristic part can be other parts of the upper body except the face. The second characteristic part can be the person's face.

[0067] In this embodiment, image segmentation processing can be performed on the sample original image, that is, removing the content of the second feature part and retaining the content of the first feature part to obtain a sample segmented image. Then, based on the sample reference image, the sample original image, and the sample segmented image, an image generation model is used to generate a sample reconstructed image corresponding to the current training perspective. Specifically, the image generation model can be used to extract key object features (i.e., second sample feature information) of the second sample object used for migration to the second feature part from the sample reference image. Feature position information corresponding to each key feature in the second feature part (i.e., sample feature position information) is extracted from the sample original image, and key object features (i.e., first sample feature information) of the first sample object in the first feature part are extracted from the sample segmented image. The sample feature position information is then used as a position constraint for migrating the second sample feature information, and the second sample feature information is fused with the first sample feature information to generate a sample reconstructed image corresponding to the current training perspective. Because the sample feature position information is used as a position constraint during the feature information fusion process, the accuracy of the migration position of the key object features of the second sample object is guaranteed, thereby improving the accuracy of the reconstructed image generation.

[0068] For example, Figure 2 As shown, the image generation model of this embodiment may further include a first feature extraction network 21, a second feature extraction network 22, a third feature extraction network 23, and an image generation network 24. The method for generating a sample reconstructed image corresponding to the current training perspective based on the above networks may be: using the first feature extraction network 21 to extract sample feature position information corresponding to the second feature part from the sample original image; using the second feature extraction network 22 to extract first sample feature information corresponding to the first feature part from the sample segmented image; using the third feature extraction network 23 to extract second sample feature information corresponding to the second feature part from the sample reference image; and using the image generation network 24 to generate a sample reconstructed image corresponding to the current training perspective based on the sample feature position information, the first sample feature information, and the second sample feature information.

[0069] Specifically, the sample original image can be input into the first feature extraction network 21, and the first feature extraction network 21 extracts the key position features of the first sample object for the second feature part (i.e., the position information corresponding to the key object features at the second feature part) from the sample original image as the sample feature position information. The sample segmentation image is input into the second feature extraction network 22, and the second feature extraction network 22 extracts the key object features of the first sample object for the first feature part from the sample segmentation image as the first sample feature information. The sample reference image is input into the third feature extraction network 23, and the third feature extraction network 23 extracts the key object features of the second sample object for the second feature part from the sample reference image as the second sample feature information. The sample feature position information, the first sample feature information, and the second sample feature information are then input into the image generation network 24. The image generation network 24 uses the sample feature position information as a position constraint, fuses the second sample feature information with the first sample feature information, and generates a sample reconstructed image.

[0070] In some embodiments, in order to improve the efficiency of model training, when training the image generation model based on the difference information between the sample reconstructed image and the sample reference image, this embodiment can only train the part of the network in the image generation model that has a larger image of the reconstructed image process. For example, only the third feature extraction network of the above-mentioned image generation model can be trained.

[0071] In some embodiments, the third feature extraction network described above in this embodiment may further include an input network layer, at least one intermediate network layer, and an output network layer. If there are multiple intermediate network layers, the multiple intermediate network layers can be connected in sequence, and each intermediate network layer will analyze and process the input information and output the sample transfer features to the next intermediate network layer. Among them, the structures of the input network layer, the intermediate network layer, and the output network layer can be the same. For example, they are all composed of a residual block (Res-Block), a self-attention (Self-Attention) module, and a cross-attention (Cross-Attention) module.

[0072] On this basis, in order to further improve the accuracy of model training, this embodiment trains the third feature extraction network of the image generation model based on the difference information between the sample reconstructed image and the sample reference image, including: obtaining the sample transfer features output by one or more intermediate network layers; using the image generation network to obtain the transfer reconstructed image corresponding to the current viewing angle to be trained based on the sample feature position information, the sample transfer feature information and the second sample feature information; based on the difference information between the sample reconstructed image and the sample reference image, and the difference information between the transfer reconstructed image and the sample reference image, the third feature extraction network of the image generation model is trained.

[0073] Specifically, if the number of intermediate network layers is one, the sample transfer feature output by the intermediate network layer can be obtained. If the number of intermediate network layers is multiple, one or more intermediate network layers can be selected from the multiple intermediate network layers, and the sample transfer feature output by the selected intermediate network layer(s) can be obtained. For example, if one intermediate network layer is selected, the intermediate network layer located at the middle position in the multiple intermediate network layers can be selected. Then, based on the above-mentioned manner of generating the sample reconstruction image, the image generation network is used to obtain the transfer reconstruction image corresponding to the current training view based on the sample feature position information, the sample transfer feature information, and the second sample feature information. Further, according to the difference information between the sample reconstruction image and the sample reference image, the first training loss is determined, and according to the difference information between the transfer reconstruction image and the sample reference image, the second training loss is determined. The third feature extraction network is trained based on the first training loss and the second training loss.

[0074] It should be noted that if multiple sample transfer features are obtained, multiple transfer reconstruction images can be obtained based on the multiple sample transfer features, and a second training loss can be determined based on the difference information between each transfer reconstruction image and the sample reference image. Alternatively, the difference information between the multiple transfer reconstruction images and the sample reference image can be integrated to determine a second training loss.

[0075] In some embodiments, in order to more realistically simulate dynamic scenes and enhance interactivity, dynamic geometric models are increasingly widely used. In order to more accurately construct a dynamic geometric model using an image generation model, the embodiment provides an implementation manner of training an image generation model in a dynamic geometric model (i.e., a dynamic first sample object geometric model) scenario, i.e., a specific implementation manner of iteratively performing the training step for multiple training views.

[0076] It should be noted that if the first sample object geometric model is a dynamic geometric model, the first sample object geometric model needs to include multiple model frames (i.e., multiple first model frames). Each first model frame can be regarded as a static geometric model of a first sample object, which corresponds to an action of the first sample object. In order to ensure the continuity of the action, the multiple first model frames have a sequence, and the sequence of the multiple first model frames is consistent with the sequence of the corresponding actions.

[0077] The specific implementation manner is as shown in Figure 3 The specific implementation manner is as shown in

[0078] S301, for the training first model frame and the current training view, the training first model frame is rendered to generate a sample original image corresponding to the current training view.

[0079] Specifically, each first model frame can be sequentially taken as a to-be-trained first model frame in order of the first model frames in the first sample object geometric model. When performing model training each time, the to-be-trained first model frame corresponding to the current training, and the current to-be-trained view, are rendered under the current to-be-trained view according to the manner similar to that described in S103 above, to obtain a rendered image under the current to-be-trained view as a sample original image corresponding to the to-be-trained first model frame under the current to-be-trained view.

[0080] S302, generating, based on the sample reference image and the sample original image, a sample reconstructed image corresponding to the current to-be-trained view by using the image generation model.

[0081] Optionally, one implementable manner of the present step can be performed in the manner of S104 above. That is, only based on the sample original image of the to-be-trained first model frame under the current to-be-trained view and the sample reference image, the sample reconstructed image corresponding to the current to-be-trained view is generated.

[0082] In some embodiments, since there is a time sequence between the multiple model frames of the dynamic geometric model, in order to ensure the time sequence consistency between the model frames in the dynamic geometric model, another implementable manner of the present step can be: based on the sample original image of the to-be-trained first model frame and at least one first model frame before the to-be-trained first model frame corresponding to the current to-be-trained view, a sample image sequence of the to-be-trained first model frame for the current to-be-trained view is constituted; based on the sample reference image and the sample image sequence, a sample reconstructed image corresponding to the to-be-trained first model frame under the current to-be-trained view is generated by using the image generation model.

[0083] Specifically, because the multiple first model frames of the first sample object geometric model are temporally sequential, when acquiring sample original images, this embodiment not only acquires the sample original image of the current first model frame to be trained at the current training viewpoint, but also acquires the sample original image of at least one first model frame preceding the first model frame to be trained at the current training viewpoint. All of these sample original images are collectively used as the sample image sequence of the first model frame to be trained at the current training viewpoint. It should be noted that if the first model frame to be trained is the first first model frame of the first sample object geometric model, the sample image sequence at this time only includes the sample original image of the first model frame to be trained at the current training viewpoint. Furthermore, when acquiring the sample original image of at least one first model frame preceding the first model frame to be trained at the current training viewpoint, since the at least one previous first model frame has already been trained at multiple training viewpoints, i.e., its sample original image at the current training viewpoint has already been acquired, the previously rendered sample original image can be directly acquired. The sample original image of the first model frame to be trained at the current training viewpoint needs to be acquired by rendering the first model frame to be trained.

[0084] At this time, when using the image generation model to reconstruct a sample image, the features of the sample original image corresponding to the current training perspective of at least one first model frame prior to the first model frame to be trained and the features of the sample original image corresponding to the first model frame to be trained at the current training perspective are combined to determine the final features of the sample original image corresponding to the first model frame to be trained at the current training perspective. The image is then reconstructed by combining the features extracted from the sample reference image. In other words, when extracting the features of the current first model frame to be trained at the current training perspective, this embodiment relies only on the current and previous first model frames and is not affected by future first model frames, thereby maintaining causality in the time dimension. This further improves the temporal consistency between reconstructed images corresponding to different model frames when subsequently reconstructing images based on the image generation model.

[0085] In some embodiments, in order to improve the accuracy of image reconstruction, this embodiment uses an image generation model to generate a sample reconstructed image corresponding to the first model frame to be trained at the current viewing angle to be trained based on a sample reference image and a sample image sequence. Image segmentation processing can be performed on multiple sample original images in the sample image sequence to obtain multiple sample segmented images corresponding to the first characteristic part; a sample segmentation sequence is formed by multiple sample segmented images; and based on the sample image sequence, the sample segmentation sequence, and the sample reference image, the image generation model is used to generate a sample reconstructed image corresponding to the first model frame to be trained at the current viewing angle to be trained.

[0086] Specifically, the embodiment is to respectively perform image segmentation processing on multiple sample original images in the sample image sequence, that is, to remove the part (i.e., the second feature part) that needs to be subjected to feature migration and to retain other parts (i.e., the first feature part), to obtain sample segmentation images corresponding to the multiple sample original images in the sample image sequence respectively, and then to form a sample segmentation sequence by using the multiple sample segmentation images obtained by segmentation. When feature extraction is subsequently performed by using the image generation model, the sample image sequence is used to extract sample position information. Since the sample position information is used to constrain the position of feature migration, the features extracted from the sample image sequence play a role of position constraint when a sample reconstruction image is generated based on the sample image sequence, the sample segmentation sequence, and the sample reference image. Therefore, the position information of the features of the multiple sample original images can be extracted to determine the sample position information. Alternatively, only the position information of the features of the sample original image corresponding to the frame of the first model to be trained can be extracted as the sample position information. The features extracted from the sample segmentation sequence directly affect the consistency of the content of the image reconstruction in the time sequence. Therefore, the embodiment can sequentially extract the features of the objects in the multiple sample segmentation images in the sample segmentation sequence to determine the first sample feature information corresponding to the sample to be trained at the current training view. The process of extracting the second sample feature information of the sample reference image and generating the reconstruction image based on the sample position information, the first sample feature information, and the second sample feature information is similar to the manner described in the above embodiment, and thus will not be described herein.

[0087] For example, in combination with the image generation model (i.e., including the first feature extraction network, the second feature extraction network, the third feature extraction network, and the image generation network) shown in Figure 2 The process of generating the sample reconstruction image corresponding to the frame of the first model to be trained at the current training view by using the image generation model based on the sample image sequence, the sample segmentation sequence, and the sample reference image is described in detail, and specifically includes the following sub-steps:

[0088] Sub-step 1: The first feature extraction network 21 is used to extract the sample position information corresponding to the second feature part from the sample image sequence. The second feature part is the part other than the first feature part of the first sample object. The sample position information can be the position information of the key features of the first sample object in the second feature part at the current training view. For example, if the second feature part is the face of a person, the sample position information can be the position of the facial feature points of the person in the face at the current training view.

[0089] Specifically, this embodiment can be based on all the sample original images in the sample image sequence to extract the sample position information corresponding to the second characteristic part of the first sample object, that is, the temporal sequence between different first model frames is taken into account when extracting the sample position information; it can also be based only on the sample original image corresponding to the first model frame to be trained to extract the sample position information corresponding to the second characteristic part of the first sample object, so as to improve the efficiency of information extraction.

[0090] Sub-step 2: Extract first sample feature information corresponding to the first feature part from the sample segmentation sequence using the second feature extraction network 22. The second feature extraction network 22 may be a network built based on a 3D Casual VAE (three-dimensional causal variational autoencoder).

[0091] Specifically, the second feature extraction network 22 of this embodiment can combine the features corresponding to the first feature part in the sample original image corresponding to at least one first model frame before the first model frame to be trained at the current training perspective, and the features corresponding to the first feature part in the sample original image corresponding to the first model frame to be trained at the current training perspective, to determine the first sample feature information corresponding to the first feature part of the first model frame to be trained at the current training perspective.

[0092] Sub-step 3: extracting second sample feature information corresponding to the second feature part from the sample reference image using the third feature extraction network 23. The specific implementation has been introduced in the above embodiment and will not be repeated here.

[0093] Sub-step 4: Using the image generation network 24, based on the sample feature position information, the first sample feature information, and the second sample feature information, a sample reconstructed image corresponding to the first model frame to be trained at the current training perspective is generated. The specific implementation method has been introduced in the above embodiment and will not be repeated here.

[0094] S303 , training the image generation model based on the difference information between the sample reconstructed image and the sample reference image, until a sample reconstructed image meeting the training requirements is obtained by using the image generation model.

[0095] Optionally, this embodiment can train all networks in the image generation model based on the difference information between the sample reconstructed image and the sample reference image. To improve model training efficiency, this embodiment can also train the third feature extraction network of the image generation model based on the difference information between the sample reconstructed image and the sample reference image. The specific implementation method has been described in the above embodiment and will not be repeated here.

[0096] Optionally, the third feature extraction network comprises an input network layer, at least one intermediate network layer, and an output network layer; at this time, in order to further improve the model training accuracy, the embodiment trains the third feature extraction network of the image generation model based on the difference information between the sample reconstructed image and the sample reference image, comprising: obtaining sample transmission features output by one or more intermediate network layers; using the image generation network, based on the sample feature position information, the sample transmission feature information, and the two sample feature information, obtaining a transmission reconstructed image corresponding to the to-be-trained first model frame at the current to-be-trained view; based on the difference information between the sample reconstructed image and the sample reference image, and the difference information between the transmission reconstructed image and the sample reference image, training the third feature extraction network of the image generation model. The specific implementation manner has been described in detail in the above embodiment.

[0097] S304, mapping the sample reconstructed image meeting the training requirement to the to-be-trained first model frame, updating the to-be-trained first model frame, and performing the training step for the to-be-trained first model frame and the next to-be-trained view, and after the to-be-trained first model frame performs the training step for all the to-be-trained views, performing the training step for the next model frame in the plurality of first model frames and the current to-be-trained view.

[0098] Specifically, after updating the to-be-trained first model frame, the updated to-be-trained first model frame is used to participate in the above training step corresponding to the next view, that is, the next view which has not been trained in the plurality of views is taken as the current to-be-trained view, and the operation of returning to S301-S304 is continued until the to-be-trained first model frame is trained for all the views, at this time, it is indicated that the training for the to-be-trained first model frame is completed, and the next model frame in the plurality of first model frames is taken as the to-be-trained first model frame, and the operation of returning to S301-S304 is continued. That is, the embodiment needs to train each first model frame based on any to-be-trained view, and the training manner is similar to the manner introduced in the above embodiment S102-S105, which will not be described here.

[0099] In the embodiment, when the image generation model is trained in the dynamic geometric model scene, each first model frame in the first sample object geometric model is sequentially trained based on a plurality of to-be-trained perspectives, and after each to-be-trained perspective training meets the requirements, the first model frame is updated according to the sample reconstruction image that meets the requirements under the to-be-trained perspective, and then the training of the next to-be-trained perspective is performed, thereby ensuring the spatial consistency of the reconstruction images generated based on each first model frame under different perspectives. In addition, for the plurality of first model frames with a time sequence relationship, when the object features (such as the first sample feature information) of the first sample object are extracted, only the current and previous first model frames are relied on, and the future first model frames are not affected, thereby maintaining the causality in the time dimension and ensuring the time sequence consistency between the reconstruction images generated based on the plurality of first model frames under the same perspective. Further, when the image generation model trained by the embodiment is used to construct a target object geometric model, the accuracy of the dynamic target object geometric model under a plurality of perspectives is improved.

[0100] Figure 4 A flowchart of an embodiment of the geometric model generation method provided in the present application, the technical solution of the embodiment can be executed by a processing end, Figure 4 The geometric model generation method can include the following steps:

[0101] S401, a plurality of target original images containing a first object corresponding to a plurality of perspectives and a target reference image containing a second object corresponding to a preset perspective are obtained.

[0102] The plurality of perspectives are obtained based on the preset perspective adjustment. The specific obtaining method has been described in the above embodiment. The first object and the second object are similar to the first sample object and the second sample object described in the above embodiment. In order to ensure the accuracy of the reconstruction image of the image generation model, the types of the first object and the second object in the embodiment can be the same as the types of the first sample object and the second sample object used in the training stage of the image generation model.

[0103] Optionally, the way of obtaining the target reference image containing the second object corresponding to the preset perspective is similar to the way of obtaining the sample reference image containing the second sample object corresponding to the preset perspective in the training of the image generation model, which will not be described here.

[0104] Optionally, one implementation of the step of obtaining the plurality of target original images containing the first object corresponding to the plurality of perspectives can be: for the first object, drawing images containing the first object from a plurality of perspectives as the plurality of target original images. Alternatively, the images containing the first object can be photographed by a camera under a plurality of perspectives as the plurality of target original images.

[0105] If a geometric model has been pre-constructed for the first object (i.e., the first object geometric model), to improve the efficiency of acquiring the target original image, this embodiment may also acquire multiple target original images containing the first object, generated by rendering the first object geometric model at multiple perspectives. The specific implementation method is similar to the method of rendering the first sample object geometric model to generate sample original images corresponding to the current training perspective during the training of the aforementioned image generation model, and is not further described here.

[0106] In some embodiments, if the first object geometric model is a dynamic geometric model, that is, the first object geometric model includes multiple second model frames, then to ensure the comprehensiveness and temporal consistency of the acquired target original images, this embodiment acquires multiple target original images containing the first object generated by rendering the first object geometric model at multiple perspectives, including: acquiring the first object geometric model; and, for any second model frame of the first object geometric model, acquiring multiple target original images containing the first object generated by rendering the second model frame at multiple perspectives. In other words, to ensure the integrity and temporal consistency of the subsequently constructed target object model, each second model frame of the first object geometric model needs to be rendered at multiple different perspectives to obtain multiple target original images corresponding to each second model frame at the multiple perspectives.

[0107] S402 : Based on a plurality of target original images and a target reference image, an image generation model is used to generate target reconstructed images corresponding to a plurality of perspectives.

[0108] The image generation model is trained using multiple sample original images containing a first sample object corresponding to multiple training perspectives and a sample reference image containing a second sample object corresponding to a preset perspective. A sample reconstructed image meeting the training requirements obtained from any training perspective is mapped into the first sample object geometric model to update the first sample object geometric model. A sample original image corresponding to any training perspective is rendered based on the updated first sample object geometric model. The specific training method has been described in the above embodiment and will not be repeated here.

[0109] Optionally, multiple target original images and target reference images are input into the image generation model at the same time. The image generation model will extract features from the multiple target original images and target reference images, and perform an image reconstruction operation based on the extracted features of each target original image combined with the features extracted from the target reference image to obtain a target reconstructed image corresponding to each target original image, that is, the features of all or part of each target original image are transferred to the features of the corresponding parts in the target reference image.

[0110] Optionally, in order to further improve the accuracy of reconstructed image generation, this embodiment may be to perform image segmentation processing on multiple target original images in sequence to obtain multiple target segmentation images corresponding to the first characteristic part; based on the target reference image, multiple target original images, and multiple target segmentation images, use the image generation model to generate target reconstructed images corresponding to multiple perspectives. For example, for any perspective, based on the target reference image, and the target original image and target segmentation image corresponding to the perspective, use the image generation model to generate the target reconstructed image corresponding to the perspective. Exemplarily, combined Figure 2 The network structure of the image generation model shown introduces the generation process of the target reconstructed image. For any perspective, the first feature extraction network 21 can be used to extract the target feature position information corresponding to the second feature part from the target original image corresponding to the perspective; the second feature extraction network 22 can be used to extract the first target feature information corresponding to the first feature part from the target segmentation image corresponding to the perspective; the third feature extraction network 23 can be used to extract the second target feature information corresponding to the second feature part from the target reference image; and the image generation network 24 can be used to generate the target reconstructed image corresponding to the perspective based on the target feature position information, the first target feature information, and the second target feature information. It should be noted that the method of generating the target reconstructed image using the image generation model in this embodiment is similar to the method of generating the sample reconstructed image corresponding to the current perspective to be trained using the image generation model based on the sample reference image, the sample original image, and the sample segmentation image introduced during the training of the above-mentioned image generation model, and will not be repeated here.

[0111] S403: Generate a target object geometric model based on target reconstructed images corresponding to multiple perspectives.

[0112] Optionally, this embodiment can generate a target object geometric model based on multiple target reconstructed images generated by the image generation model. For example, a three-dimensional Gaussian sputtering technique can be used to construct a three-dimensional Gaussian model as the target object geometric model based on multiple target reconstructed images.

[0113] If the first object geometric model already exists, this embodiment can also map the target reconstructed images corresponding to multiple viewpoints onto the first object geometric model to obtain the target object geometric model. The specific process is similar to the method for updating the first sample object geometric model based on the sample reconstructed images described in the image generation model training phase and is not further described here.

[0114] In the embodiment of the present application, the image generation model training phase sequentially trains the image generation model for multiple viewpoints. This helps the image generation model better learn the algorithm for image reconstruction at each viewpoint, compared to training the model simultaneously for multiple viewpoints, thereby improving the accuracy of the reconstructed image generated for each viewpoint. Furthermore, after completing training for each viewpoint, the embodiment of the present application also updates the first sample object geometric model based on the sample reconstructed image generated when the training requirements are met, and renders the sample original image for the next viewpoint training based on the updated first sample object geometric model. This helps the image generation model learn the spatial correspondence between the reconstructed image at that viewpoint and other viewpoints during training for any viewpoint, thereby improving the spatial consistency of the reconstructed images generated from different viewpoints. Therefore, the image generation model trained in this manner in this embodiment, when constructing the target object geometric model based on multiple target original images containing the first object and a target reference image containing the second object corresponding to a preset viewpoint, can well consider the spatial consistency of the target object geometric model at different viewpoints, thereby improving the accuracy of the target object geometric model at multiple viewpoints.

[0115] In some embodiments, if multiple target original images are rendered based on multiple second model frames of the first object geometric model, in order to improve the accuracy of the subsequently generated reconstructed image, this step can be for any perspective, and the target original images of the corresponding perspective of the current second model frame and at least one second model frame before the current second model frame constitute the target image sequence for the perspective of the current second model frame; based on the target image sequence and the target reference image, the image generation model is used to generate the target reconstructed image corresponding to the current second model frame under the perspective.

[0116] It should be noted that the first object geometric model at this time is a dynamic model, so there is a time sequence between the multiple second model frames. In this embodiment, each second model frame can be used as the current second model frame in turn, and the current second model frame and at least one second model frame before it correspond to the target original image of the view to construct a target image sequence. That is, for each second model frame, it corresponds to a target image sequence at each view. If the second model frame is the first second model frame of the first object geometric model, then the corresponding target image sequence only has one target original image corresponding to the second model frame. Based on the target image sequence and the target reference image, the image generation model is used to generate the target reconstructed image corresponding to the current second model frame at the view angle. The method of using the image generation model to generate the sample reconstructed image corresponding to the first model frame to be trained at the current view angle to be trained based on the sample reference image and the sample image sequence is similar to the method introduced in the above-mentioned image generation model training using the image generation model, and based on the sample reference image and the sample image sequence, the image generation model is used. It will not be repeated here.

[0117] On this basis, if the target geometric model is a dynamic model, the method of performing image segmentation processing on the target original image and then reconstructing the image can be: for any current second model frame, performing image segmentation processing on multiple target original images in the target image sequence corresponding to the current second model frame, and obtaining multiple target segmentation images corresponding to the first characteristic part; forming a target segmentation sequence from multiple target segmentation images; based on the target image sequence, the target segmentation sequence, and the target reference image, using the image generation model, generate the target reconstructed image corresponding to the current second model frame under the said perspective.

[0118] The specific implementation method is similar to the method introduced in the above-mentioned image generation model training, which is to perform image segmentation processing on multiple sample original images in the sample image sequence respectively to obtain multiple sample segmentation images corresponding to the first characteristic part; the sample segmentation sequence is composed of multiple sample segmentation images; based on the sample image sequence, the sample segmentation sequence, and the sample reference image, the image generation model is used to generate a sample reconstructed image corresponding to the first model frame to be trained at the current training perspective, which will not be repeated here.

[0119] At this time, combined Figure 2 The network structure of the image generation model shown introduces the generation process of the target reconstructed image. Using the first feature extraction network, the target position information corresponding to the second feature part is extracted from the target image sequence; the second feature part is the other part of the first object except the first feature part; using the second feature extraction network, the first target feature information corresponding to the first feature part is extracted from the target segmentation sequence; using the third feature extraction network, the second target feature information corresponding to the second feature part is extracted from the target reference image; using the image generation network, based on the target position information, the first target feature information and the second target feature information, the target reconstructed image corresponding to the current second model frame under the viewing angle is generated. The specific implementation method has been introduced in the image generation model training stage of the above embodiment and will not be repeated here.

[0120] If the first object geometric model includes multiple second model frames, the target object geometric model can be generated based on target reconstructed images corresponding to multiple viewpoints by mapping multiple target reconstructed images corresponding to any second model frame at multiple viewpoints to the second model frame to obtain a target model frame corresponding to the second model frame; and the target object geometric model is constructed based on the target model frames corresponding to each of the multiple second model frames. Specifically, for each second model frame, after mapping the target reconstructed images from different viewpoints to the second model frame, the second model frame now has object features that have been transferred to the second object at multiple viewpoints. The multiple second model frames can then be combined according to their temporal sequence to obtain a dynamic target object geometric model.

[0121] In the dynamic geometric model scene, whether in the training stage of the image generation model or the inference stage (i.e., the stage of constructing the target object model by using the image generation model), for each model frame in the dynamic geometric model, training or inference is sequentially performed based on multiple perspectives, thereby ensuring the spatial consistency of the reconstructed images generated based on each model frame under different perspectives. In addition, for multiple model frames with a time sequence relationship, when extracting sample feature information, only the current and previous model frames are relied on, and the future model frames are not affected, thereby maintaining the causality in the time dimension and ensuring the time sequence consistency between the reconstructed images generated based on multiple model frames under the same perspective. Furthermore, the accuracy of the dynamic target object geometric model constructed by using the image generation model under multiple perspectives is improved.

[0122] In some embodiments, the first sample object and the second sample object of the embodiments of the present application can be a human upper body image, the first sample object geometric model is a dynamic three-dimensional Gaussian model, and then the method for training the image generation model in the actual application scenario will be introduced as follows: Figure 5

[0123] Since the first sample object Gaussian model is a dynamic three-dimensional Gaussian model, the first model frame of the first sample object Gaussian model can be selected as a to-be-trained first model frame, and a current to-be-trained perspective can be selected from multiple to-be-trained perspectives. For example, the first model frame of the first sample object Gaussian model is selected as the to-be-trained first model frame, and a first to-be-trained perspective is selected from the multiple to-be-trained perspectives. Figure 5 ​As shown, assuming that the current training perspective is a frontal perspective and the preset perspective is also a frontal perspective, a second sample subject face image (i.e., a sample reference image) 52 corresponding to the frontal perspective (i.e., the preset perspective) is obtained. The first model frame to be trained is rendered for the frontal perspective (i.e., the current training perspective) to generate a first sample subject upper body image (i.e., a sample original image) 50 corresponding to the frontal perspective. The first sample subject upper body image 50 and at least one sample original image rendered at the current training perspective, which precedes the first model frame to be trained, are used as a sample image sequence for the frontal training perspective of the first model frame to be trained. Image segmentation processing is performed on each of the multiple sample original images in the sample image sequence to obtain multiple sample segmented images. The multiple segmented sample segmented images form a sample segmentation sequence 51. Then, using a first feature extraction network, sample position information corresponding to facial parts is extracted from the first sample subject upper body image 50. A second feature extraction network, such as a 3D Casual VAE (three-dimensional causal variational autoencoder), is used to determine the final first sample feature information corresponding to other parts of the first model frame to be trained, excluding the face, based on the feature information of the sample segmentation images of the first model frame to be trained and the first model frames before it in the sample segmentation sequence 51. The first model frames after the first model frame to be trained are not considered, so that the extracted first sample feature information can well follow the temporal causality. A third feature extraction network is used to extract second sample feature information corresponding to facial parts from the second sample object face image 52. Specifically, the third feature extraction network of this embodiment can be composed of an input network layer with the same structure (i.e., a residual block (Res-Block), a self-attention (Self-Attention) module, and a cross-attention (Cross-Attention) module), multiple intermediate network layers, and an output network layer. The output of the input network layer is the second sample feature information, and the output of the intermediate network layer is the sample transfer feature, which is passed to the next network layer (such as the next intermediate network layer or the output network layer).

[0124] Then, based on the sample feature position information, the first sample feature information, and the second sample feature information, a reconstructed image is generated by using an image generation network. Specifically, the image generation network at this time can be a diffusion model constructed by multiple Block (a basic unit for constructing a network) of Transformer (a deep learning model based on self-attention mechanism) structure. At this time, the first sample feature information can be input as the main network of the diffusion model, and the sample position information and the third feature extraction network are respectively input into the corresponding Transformer Block. The reconstructed image is generated by the diffusion model based on the input of the three kinds of information. Since the reconstructed image at this time still has noise, it needs to be processed by a de-noising network to obtain the sample reconstructed image 53 corresponding to the first model frame to be trained under the front view.

[0125] In order to improve the accuracy of image generation model training, the embodiment can obtain sample transmission features of the intermediate network layer output located at the intermediate position, and then obtain a transmission reconstructed image corresponding to the first model frame to be trained under the front view based on the sample feature position information, the sample transmission feature information, and the second sample feature information by using the image generation network. The training loss 1 is determined based on the difference information between the sample reconstructed image 53 and the second sample object face image 52, and the training loss 2 is determined based on the difference information between the transmission reconstructed image and the second sample object face image 52. The third feature extraction network of the image generation model is trained based on the training loss 1 and the training loss 2 until the sample reconstructed image that meets the training requirements is obtained. Then, the sample reconstructed image that meets the training requirements is mapped to the first model frame to be trained to update the first model frame to be trained. The updated first model frame to be trained is used to participate in the training step corresponding to the next view until the first model frame to be trained is trained for multiple views. The next model frame in the multiple first model frames is taken as the first model frame to be trained, and the above training step is continued to be executed.

[0126] For example, the training loss (i.e., the training loss 1 and the training loss 2) constructed by the embodiment is shown in formula 1.

[0127]

[0128] Wherein, is the training loss; E is the mean square error loss function; is the reconstructed image generated after the tth iteration training of the image generation network; t is the tth iteration training; C is the second sample feature information extracted by the third feature extraction network; C i is the sample feature position information and the second sample feature information; ∈ is the second sample object face image 52; ∈θ is the sample reconstructed image 53; is the feature distribution range.

[0129] The embodiment sequentially trains each first model frame in the first sample object geometry model based on multiple perspectives, and after each perspective training meets the requirements, the first model frame is updated according to the sample reconstruction image that meets the requirements under the perspective, and then the next perspective training is performed, which guarantees the spatial consistency of the reconstruction images generated under different perspectives based on each first model frame. In addition, for multiple first model frames with a time sequence relationship, when extracting the object features (such as first sample feature information) of the first sample object, only the current and previous first model frames are relied on, and the future first model frames are not affected, thereby maintaining the causality in the time dimension and guaranteeing the time sequence consistency between the reconstruction images generated under the same perspective based on multiple first model frames. In addition, the embodiment calculates the loss based on the features output by the output layer and the intermediate layer of the third feature extraction network to train the third feature extraction network, so that the facial features of the reconstruction image are more similar to the features of the input second sample object face image 52, thereby improving the accuracy of the image generation model training.

[0130] Next, combined with Figure 6 The inference process of the trained image generation model in the actual scene (that is, editing the upper body image of the first object Gaussian model into a Gaussian model containing the second object face features) is introduced.

[0131] First, the second object face image corresponding to the front perspective (that is, the target reference image) 62 is obtained, and the dynamic Gaussian model of the upper body image of the first object (containing multiple second model frames) is obtained, and the first object upper body image corresponding to each second model frame under multiple perspectives (that is, the target original image) 60 is rendered. For each second model frame under each perspective, the target original image corresponding to the second model frame and at least one second model frame before it under the perspective is taken as the target image sequence of the second model frame under the perspective, and the multiple target original images in the target image sequence are subjected to face segmentation processing to obtain a target segmentation sequence 61 containing multiple target segmentation images.

[0132] Then the first object upper body image 60 corresponding to each second model frame under multiple perspectives is input to the first feature extraction network, target position information corresponding to the face region of each second model frame under multiple perspectives is obtained, the target segmentation sequence 61 corresponding to each second model frame under multiple perspectives is input to the second feature extraction network, first target feature information corresponding to other regions except the face region of each second model frame under multiple perspectives is obtained; the second object face image 62 is input to the third feature extraction network, and the first target feature information corresponding to the face region of the second sample object is obtained; then, the image generation network is used to generate a reconstructed image based on the obtained target position information, the first target feature information and the second target feature information; at this time, the reconstructed image has noise, and needs to be processed through the denoising network to obtain the final target reconstructed image 63 corresponding to each second model frame under different perspectives. Finally, the multiple target reconstructed images corresponding to each second model frame under multiple perspectives are mapped into the corresponding second model frame to obtain the target model frame corresponding to each second model frame; and based on the target model frame corresponding to each second model frame, a target object Gaussian model is constructed.

[0133] The embodiment is based on a target reference image (i.e. the second object face image 62) and an existing dynamic Gaussian model (i.e. the dynamic Gaussian model of the first object upper body image), that is, the target object Gaussian model containing the characteristics of the target reference image can be constructed by using the image generation model. In the image generation model training process, the migration of the second feature information is constrained by the target position information, and the model training stage is trained separately for different perspectives, so that the target object Gaussian model generated by the embodiment ensures consistency under multiple perspectives. In addition, when the first target feature information is extracted based on the second feature extraction network in the embodiment, the time sequence causality between the multiple second model frames is considered, which greatly reduces the time dimension jitter and flicker problem, and improves the consistency of the finally constructed target object Gaussian model in the spatial dimension and the time dimension.

[0134] The detailed implementation and beneficial effects of each step in the method of the embodiment have been described in detail in the foregoing embodiments, and will not be described in detail here.

[0135] It should be noted that in some of the processes described in the above embodiments and the accompanying drawings, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or may be executed in parallel. The serial numbers of the operations, such as 101, 102, etc., are only used to distinguish between different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit "first" and "second" to being different types.

[0136] Figure 7 This is a schematic diagram of the structure of a model training device provided by an exemplary embodiment of the present application. The device is applied to the training of an image generation model, and the device includes:

[0137] A first acquisition module 701 is configured to acquire a geometric model of a first sample object and a sample reference image containing a second sample object corresponding to a preset viewing angle;

[0138] A perspective determination module 702 is configured to determine a plurality of perspectives to be trained based on the preset perspectives;

[0139] The model training module 703 is used to perform training steps for the multiple perspectives to be trained, and the training steps include: for the current perspective to be trained, rendering the first sample object geometric model to generate a sample original image corresponding to the current perspective to be trained; based on the sample reference image and the sample original image, using the image generation model to generate a sample reconstructed image corresponding to the current perspective to be trained; based on the difference information between the sample reconstructed image and the sample reference image, training the image generation model until a sample reconstructed image that meets the training requirements is obtained using the image generation model; mapping the sample reconstructed image that meets the training requirements to the first sample object geometric model, updating the first sample object geometric model, and performing training steps for the next perspective to be trained.

[0140] In some embodiments, the first sample object geometric model includes multiple first model frames; the model training module 703 is specifically used to render the first model frame to be trained to generate a sample original image corresponding to the current view to be trained for the first model frame to be trained and the current view to be trained; the model training module 703 is also specifically used to map the sample reconstructed image that meets the training requirements to the first model frame to be trained, update the first model frame to be trained, and execute training steps for the first model frame to be trained and the next view to be trained, and after the training steps for the first model frame to be trained are all executed for the multiple view angles to be trained, execute training steps for the next model frame among the multiple first model frames and the current view angle to be trained.

[0141] In some embodiments, the model training module 703 is further specifically used to construct a sample image sequence of the first model frame to be trained for the current perspective to be trained based on the sample original image of the first model frame to be trained and at least one first model frame before the first model frame to be trained corresponding to the current perspective to be trained; based on the sample reference image and the sample image sequence, using an image generation model, generate a sample reconstructed image corresponding to the first model frame to be trained at the current perspective to be trained.

[0142] In some embodiments, the model training module 703 is further specifically used to perform image segmentation processing on multiple sample original images in the sample image sequence to obtain multiple sample segmentation images corresponding to the first characteristic part; the multiple sample segmentation images constitute a sample segmentation sequence; based on the sample image sequence, the sample segmentation sequence, and the sample reference image, an image generation model is used to generate a sample reconstructed image corresponding to the first model frame to be trained at the current viewing angle to be trained.

[0143] In some embodiments, the image generation model includes: a first feature extraction network, a second feature extraction network, a third feature extraction network and an image generation network; the model training module 703 is also specifically used to use the first feature extraction network to extract sample position information corresponding to the second feature part from the sample image sequence; the second feature part is other parts of the first sample object geometric model except the first feature part; using the second feature extraction network, the first sample feature information corresponding to the first feature part is extracted from the sample segmentation sequence; using the third feature extraction network, the second sample feature information corresponding to the second feature part is extracted from the sample reference image; using the image generation network, based on the sample feature position information, the first sample feature information and the second sample feature information, a sample reconstructed image corresponding to the first model frame to be trained at the current training perspective is generated.

[0144] In some embodiments, the model training module 703 is further specifically used to train the third feature extraction network of the image generation model based on the difference information between the sample reconstructed image and the sample reference image.

[0145] In some embodiments, the third feature extraction network includes an input network layer, at least one intermediate network layer, and an output network layer; the model training module 703 is also specifically used to obtain sample transfer features output by one or more intermediate network layers; using the image generation network, based on the sample feature position information, the sample transfer feature information and the second sample feature information, the transfer reconstructed image corresponding to the first model frame to be trained at the current training perspective is obtained; based on the difference information between the sample reconstructed image and the sample reference image, and the difference information between the transfer reconstructed image and the sample reference image, the third feature extraction network of the image generation model is trained.

[0146] Figure 7 The model training device can perform Figure 1 or Figure 4 The model training method described in the illustrated embodiment, its implementation principle and technical effects are not repeated here. The specific manner in which each module and unit performs operations in the 7 devices in the above embodiment has been described in detail in the embodiment of the method, and will not be elaborated here.

[0147] Figure 8 A schematic structural diagram of a geometric model generation device provided in an exemplary embodiment of the present application, the device comprising:

[0148] The second acquisition module 801 is configured to acquire a plurality of target original images containing a first object corresponding to a plurality of viewing angles, and a target reference image containing a second object corresponding to a preset viewing angle;

[0149] A second model running module 802 is configured to generate target reconstructed images corresponding to the multiple perspectives respectively based on the multiple target original images and the target reference image using an image generation model;

[0150] A model generation module 803 is configured to generate a geometric model of the target object based on the target reconstructed images corresponding to the multiple perspectives;

[0151] In which, the image generation model is trained using multiple sample original images containing the first sample object corresponding to the multiple perspectives to be trained and the sample reference image containing the second sample object corresponding to the preset perspective; the sample reconstructed image that meets the training requirements obtained at any perspective to be trained is used to map to the first sample object geometric model to update the first sample object geometric model; the sample original image corresponding to any perspective to be trained is rendered based on the updated first sample object geometric model.

[0152] In some embodiments, the second acquisition module 801 is specifically used to acquire multiple target original images containing the first object generated by rendering the first object geometric model at multiple perspectives; and acquire a target reference image containing the second object corresponding to a preset perspective.

[0153] The model generation module 803 is specifically configured to map the target reconstructed images corresponding to the multiple perspectives into the first object geometric model to obtain a target object geometric model.

[0154] In some embodiments, the first object geometric model includes multiple second model frames; the second acquisition module 801 is further specifically used to obtain the first object geometric model; for any second model frame of the first object geometric model, obtain multiple target original images containing the first object generated by rendering the second model frame under multiple perspectives.

[0155] In some embodiments, the second model operation module 802 is specifically used to: for any perspective, form a target image sequence of the current second model frame for the perspective by the current second model frame and at least one second model frame before the current second model frame, which corresponds to the target original image of the perspective; and use an image generation model to generate a target reconstructed image corresponding to the current second model frame at the perspective based on the target image sequence and the target reference image.

[0156] In some embodiments, the second model operation module 802 is further specifically used to: perform image segmentation processing on multiple target original images in the target image sequence to obtain multiple target segmentation images corresponding to the first characteristic part; form a target segmentation sequence from the multiple target segmentation images; based on the target image sequence, the target segmentation sequence, and the target reference image, use the image generation model to generate the target reconstructed image corresponding to the current second model frame under the said perspective.

[0157] In some embodiments, the image generation model includes: a first feature extraction network, a second feature extraction network, a third feature extraction network and an image generation network; the second model operation module 802 is also specifically used to: use the first feature extraction network to extract the target position information corresponding to the second feature part from the target image sequence; the second feature part is other parts of the first object except the first feature part; use the second feature extraction network to extract the first target feature information corresponding to the first feature part from the target segmentation sequence; use the third feature extraction network to extract the second target feature information corresponding to the second feature part from the target reference image; use the image generation network to generate the target reconstructed image corresponding to the current second model frame under the viewing angle based on the target position information, the first target feature information and the second target feature information.

[0158] In some embodiments, the model generation module 803 is further specifically used to map multiple target reconstructed images corresponding to any second model frame at multiple perspectives to the second model frame to obtain a target model frame corresponding to the second model frame; and construct a target object geometric model based on the target model frames corresponding to each of the multiple second model frames.

[0159] Figure 8 The geometric model generating device can execute Figure 4 The implementation principle and technical effects of the geometric model generation method described in the illustrated embodiment will not be described in detail. The specific manner in which each module and unit performs operations in the 8 devices in the above embodiment has been described in detail in the embodiment of the method, and will not be elaborated here.

[0160] Figure 9 This is a schematic diagram of a computing device according to an embodiment of the present application. Figure 9 As shown, in practice, the computing device may include: a storage component 901 and a processing component 902 .

[0161] The storage component 901 is used to store computer programs and can be configured to store various other data to support operations on the computing device. Examples of such data include instructions for any application or method operating on the computing device, data structures, contact data, phone book data, messages, images, videos, etc.

[0162] The processing component 902 is coupled to the storage component 901 and is used to execute the computer program in the storage component 901 to implement the following Figure 1 and Figure 3 The model training method shown in Figure 4 The geometric model generation method shown.

[0163] Further, if Figure 9 As shown, the computing device may further include: a communication component 903, a display component 904, a power component 905, an audio component 906 and other components. Figure 9 Only some components are shown schematically, which does not mean that the equipment only includes Figure 9 In addition, Figure 9 The components in the dotted box are optional components, not mandatory components, and depend on the specific product form of the computing device. The computing device of this embodiment can be implemented as a terminal device such as a desktop computer, a laptop computer, a smart phone or an IOT (Internet of Things) device, or a server device such as a conventional server, a cloud server or a server array. If the computing device of this embodiment is implemented as a terminal device such as a desktop computer, a laptop computer, a smart phone, etc., it can include Figure 8 If the computing device of this embodiment is implemented as a server device such as a conventional server, a cloud server or a server array, it may not include the components in the dotted box; Figure 9 Components within the dotted box.

[0164] The processing component includes one or more processors to execute computer instructions to perform all or part of the steps in the above method. Of course, the processing component can also be implemented as one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above method.

[0165] The above-mentioned storage component can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random-access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0166] The communication component is configured to facilitate wired or wireless communication between the device in which the communication component resides and other devices. The device in which the communication component resides may access a wireless network based on a communication standard, such as a mobile communication network, or a combination thereof. In an exemplary embodiment, the communication component receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel.

[0167] The display assembly may include a screen, which may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touches, slides, and gestures on the touch panel. The touch sensor may not only sense the boundaries of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0168] The power supply assembly provides power to various components of the device in which the power supply assembly is located. The power supply assembly may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device in which the power supply assembly is located.

[0169] The above-mentioned audio component can be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC), and when the device where the audio component is located is in an operating mode, such as call mode, recording mode, and voice recognition mode, the microphone is configured to receive external audio signals. The received audio signal can be further stored in a memory or sent via a communication component. In some embodiments, the audio component also includes a speaker for outputting audio signals.

[0170] Accordingly, the embodiments of the present application also provide a computer readable storage medium storing a computer program, when the computer program is executed by a processor, the processor is enabled to implement each step in the above-mentioned method embodiments. The computer readable storage medium includes volatile or non-volatile or a combination thereof, and can be removable or non-removable. Examples of the computer readable storage medium include, but are not limited to, phase-change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital video disc (DVD) or other optical storage, magnetic cassette, magnetic tape disk storage or other magnetic storage devices, or any other non-transmission medium

[0171] Accordingly, the embodiments of the present application also provide a computer program product, the computer program product includes a computer program or instructions, when the computer program or instructions are executed by a processor, the processor is enabled to implement each step in the above-mentioned method embodiments. It should be understood that each process or a combination of multiple processes in the above-mentioned method flow can be implemented by the computer program or instructions. In addition, these computer programs or instructions can be applied to the processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing devices, so that the processor of the general-purpose computer, the special-purpose computer, the embedded processor or other programmable data processing devices can be implemented as a device for implementing the corresponding functions in the above-mentioned method embodiments.

[0172] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-mentioned system, device and unit can refer to the corresponding process in the above-mentioned method embodiments, which will not be described here.

[0173] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0174] Finally, it should be noted that the above are merely examples of the present application and are not intended to limit the present application. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application are intended to be included within the scope of the claims of the present application.

Claims

1. A model training method, characterized in that: Applied to the training of image generation models, including: Acquire a geometric model of a first sample object and a sample reference image containing a second sample object corresponding to a preset viewing angle; Based on the preset perspective, determining a plurality of perspectives to be trained; For the multiple perspectives to be trained, a training step is performed, the training step comprising: For the current viewing angle to be trained, rendering the first sample object geometric model to generate a sample original image corresponding to the current viewing angle to be trained; Based on the sample reference image and the sample original image, using the image generation model, generate a sample reconstructed image corresponding to the current viewing angle to be trained; Based on the difference information between the sample reconstructed image and the sample reference image, the image generation model is trained until a sample reconstructed image that meets the training requirements is obtained by using the image generation model; The sample reconstructed image that meets the training requirements is mapped to the first sample object geometric model, the first sample object geometric model is updated, and the training step is performed for the next viewing angle to be trained.

2. The method according to claim 1, characterized in that The first sample object geometric model includes a plurality of first model frames; The rendering of the first sample object geometric model to generate the sample original image corresponding to the current viewing angle to be trained includes: For the first model frame to be trained and the current viewing angle to be trained, rendering the first model frame to be trained to generate a sample original image corresponding to the current viewing angle to be trained; The step of mapping the sample reconstructed image that meets the training requirements to the first sample object geometric model, updating the first sample object geometric model, and performing training for the next viewing angle to be trained includes: Map the sample reconstructed image that meets the training requirements to the first model frame to be trained, update the first model frame to be trained, and perform training steps on the first model frame to be trained and the next view to be trained. After the training steps are all performed on the first model frame to be trained for multiple view angles to be trained, perform training steps on the next model frame among the multiple first model frames and the current view angle to be trained.

3. The method according to claim 2, characterized in that The generating, based on the sample reference image and the sample original image, the sample reconstructed image corresponding to the current to-be-trained perspective by using an image generation model includes: Based on the first model frame to be trained and at least one first model frame before the first model frame to be trained, which corresponds to the sample original image of the current viewing angle to be trained, a sample image sequence of the first model frame to be trained for the current viewing angle to be trained is constructed; Based on the sample reference image and the sample image sequence, an image generation model is used to generate a sample reconstructed image corresponding to the first model frame to be trained at the current viewing angle to be trained.

4. The method according to claim 3, characterized in that The generating, based on the sample reference image and the sample image sequence, a sample reconstructed image corresponding to the first model frame to be trained at the current viewing angle to be trained by using an image generation model, includes: Performing image segmentation processing on a plurality of sample original images in the sample image sequence to obtain a plurality of sample segmentation images corresponding to the first characteristic part; forming a sample segmentation sequence from the plurality of sample segmentation images; Based on the sample image sequence, the sample segmentation sequence, and the sample reference image, an image generation model is used to generate a sample reconstructed image corresponding to the first model frame to be trained at the current viewing angle to be trained.

5. The method according to claim 4, characterized in that The image generation model includes: a first feature extraction network, a second feature extraction network, a third feature extraction network and an image generation network; The generating, based on the sample image sequence, the sample segmentation sequence, and the sample reference image, a sample reconstructed image corresponding to the first model frame to be trained at the current viewing angle to be trained by using an image generation model includes: Extracting sample position information corresponding to a second characteristic part from the sample image sequence using the first feature extraction network; the second characteristic part is a part of the first sample object geometric model other than the first characteristic part; extracting first sample feature information corresponding to the first feature part from the sample segmentation sequence using the second feature extraction network; extracting second sample feature information corresponding to the second feature part from the sample reference image using the third feature extraction network; The image generation network is used to generate a sample reconstructed image corresponding to the first model frame to be trained at the current viewing angle to be trained based on the sample feature position information, the first sample feature information, and the second sample feature information.

6. The method according to claim 5, characterized in that The training of the image generation model based on the difference information between the sample reconstructed image and the sample reference image includes: Based on the difference information between the sample reconstructed image and the sample reference image, the third feature extraction network of the image generation model is trained.

7. The method according to claim 6, characterized in that The third feature extraction network includes an input network layer, at least one intermediate network layer, and an output network layer; Training the third feature extraction network of the image generation model based on difference information between the sample reconstructed image and the sample reference image includes: Obtain sample transfer features output by one or more intermediate network layers; Using the image generation network, based on the sample feature position information, the sample transfer feature information, and the second sample feature information, a transfer reconstructed image corresponding to the first model frame to be trained at the current viewing angle to be trained is obtained; The third feature extraction network of the image generation model is trained based on the difference information between the sample reconstructed image and the sample reference image, and the difference information between the transferred reconstructed image and the sample reference image.

8. A method for generating a geometric model, characterized in that: include: Acquire a plurality of target original images containing a first object corresponding to a plurality of viewing angles, and a target reference image containing a second object corresponding to a preset viewing angle; The multiple viewing angles are obtained based on the preset viewing angle adjustment; Based on the multiple target original images and the target reference image, using an image generation model, generating target reconstructed images corresponding to the multiple perspectives respectively; generating a target object geometric model based on the target reconstructed images corresponding to the multiple perspectives; In which, the image generation model is trained using multiple sample original images containing a first sample object corresponding to multiple viewing angles to be trained and a sample reference image containing a second sample object corresponding to the preset viewing angle; a sample reconstructed image that meets the training requirements obtained at any viewing angle to be trained is used to map to the first sample object geometric model to update the first sample object geometric model; and the sample original image corresponding to any viewing angle to be trained is rendered based on the updated first sample object geometric model.

9. The method according to claim 8, characterized in that The acquiring of a plurality of target original images containing a first object corresponding to a plurality of perspectives and a target reference image containing a second object corresponding to a preset perspective includes: Acquire a plurality of target original images containing the first object generated by rendering the first object geometric model at multiple perspectives; Acquire a target reference image containing a second object corresponding to a preset viewing angle; Generating a target object geometric model based on target reconstructed images corresponding to the multiple perspectives includes: The target reconstructed images corresponding to the multiple perspectives are mapped to the first object geometric model to obtain a target object geometric model.

10. The method according to claim 9, characterized in that The first object geometric model includes a plurality of second model frames; The step of obtaining a plurality of target original images containing the first object and rendering the first target geometric model at multiple perspectives includes: Acquire the first object geometric model; For any second model frame of the first object geometric model, a plurality of target original images containing the first object generated by rendering the second model frame at multiple perspectives are obtained.

11. The method according to claim 10, characterized in that The generating, based on the multiple target original images and the target reference image, target reconstructed images corresponding to the multiple perspectives respectively using an image generation model includes: For any viewing angle, a target original image corresponding to the viewing angle of the current second model frame and at least one second model frame before the current second model frame constitute a target image sequence of the current second model frame for the viewing angle; Based on the target image sequence and the target reference image, an image generation model is used to generate a target reconstructed image corresponding to the current second model frame at the viewing angle.

12. The method according to claim 11, characterized in that The generating, based on the target image sequence and the target reference image, a target reconstructed image corresponding to the current second model frame at the viewing angle by using an image generation model includes: Performing image segmentation processing on the plurality of target original images in the target image sequence to obtain a plurality of target segmentation images corresponding to the first characteristic part; The plurality of target segmentation images form a target segmentation sequence; Based on the target image sequence, the target segmentation sequence, and the target reference image, an image generation model is used to generate a target reconstructed image corresponding to the current second model frame at the viewing angle.

13. The method according to claim 12, characterized in that The image generation model includes: a first feature extraction network, a second feature extraction network, a third feature extraction network and an image generation network; The generating, based on the target image sequence, the target segmentation sequence, and the target reference image, a target reconstructed image corresponding to the current second model frame at the viewing angle by using an image generation model includes: Extracting target position information corresponding to a second characteristic part from the target image sequence using the first feature extraction network; the second characteristic part is a part of the first object other than the first characteristic part; extracting first target feature information corresponding to the first feature part from the target segmentation sequence using the second feature extraction network; extracting second target feature information corresponding to the second feature part from the target reference image using the third feature extraction network; The image generation network is used to generate a target reconstructed image corresponding to the current second model frame at the viewing angle based on the target position information, the first target feature information, and the second target feature information.

14. The method according to claim 10 or 11, characterized in that Generating a target object geometric model based on target reconstructed images corresponding to the multiple perspectives includes: Mapping a plurality of target reconstructed images corresponding to any second model frame at a plurality of viewing angles to the second model frame to obtain a target model frame corresponding to the second model frame; A target object geometric model is constructed based on the target model frames corresponding to each of the plurality of second model frames.

15. A computing device, characterized in that including processing components and storage components; The storage component stores a computer program; the computer program is used to be called and executed by the processing component to implement the model training method as described in any one of claims 1 to 7, or to implement the geometric model generation method as described in any one of claims 8 to 14.

16. A computer-readable storage medium, characterized in that A computer program is stored thereon, and when the computer program is executed by the processing component, it implements the model training method according to any one of claims 1 to 7, or implements the geometric model generation method according to any one of claims 8 to 14.

17. A computer program product, characterized in that It includes a computer program or instructions, which, when executed by a processing component, implements the model training method as described in any one of claims 1 to 7, or implements the geometric model generation method as described in any one of claims 8 to 14.

Images