Three-dimensional model editing method and device, storage medium and program product

Abstract

The invention discloses a three-dimensional model editing method and device, a storage medium and a program product, and relates to the technical field of image processing, and the method comprises the steps: obtaining a plurality of source images at different visual angles, each source image comprising a target object; each target object is in a source state; determining target text prompt information corresponding to each target multi-view image to be generated; the target text prompt information comprises target state information of the target object; inputting the plurality of source images and the target text prompt information into a multi-view image editing model which is trained to be converged, and outputting a target multi-view image of the plurality of source images under the corresponding view; the target multi-view-angle image is an image of a target state formed from the source state after the target object executes a target action under the view angle corresponding to the source image. Therefore, the state of the target object is changed, and non-rigid editing is realized.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a three-dimensional model editing method, device, storage medium and program product. Background Art

[0002] Nowadays, creating 3D models plays a key role in many applications and industries. 3D models can be used in film / game production to enhance visualization and interactivity, so creating high-quality 3D models is particularly important.

[0003] In the related art, a pre-trained image editing model is used to edit the text description of the source image corresponding to the source 3D model to obtain the edited image, and then the edited image is promoted to a 3D mode by reconstruction to obtain the target 3D model, thereby achieving 3D editing. However, the above-mentioned related art can change some features of objects in the source 3D model in the same state or replace objects of the same type, but cannot process the state change of the target object. Summary of the invention

[0004] The present application provides a three-dimensional model editing method, device, storage medium and program product to solve the problem of partial feature change of objects in the same state or replacement of objects of the same type in the related art, so as to achieve the effect of changing the state of the target object.

[0005] In a first aspect, the present application provides a three-dimensional model editing method, comprising:

[0006] A plurality of source images at different viewing angles are obtained from a source three-dimensional model, where the source images are images corresponding to the three-dimensional model at different viewing angles; each source image includes a target object; and each target object is in a source state;

[0007] Determine target text prompt information corresponding to each target multi-view image to be generated; the target text prompt information includes target state information of the target object;

[0008] Input multiple source images and target text prompt information into a multi-view image editing model that has been trained to convergence, and output target multi-view images of multiple source images at corresponding viewpoints to achieve non-rigid editing; the target multi-view image is an image of a target state formed from a source state after a target object performs a target action at the viewpoint corresponding to the source image;

[0009] The target 3D model is reconstructed based on the multi-view images of each target.

[0010] In a second aspect, the present application also provides an editing device, including: a memory and a processor;

[0011] The memory stores the computer program;

[0012] When the processor executes the computer program, the steps of any one of the above three-dimensional model editing methods are implemented.

[0013] In a third aspect, the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program, when executed by a processor, implements the steps of any of the above-mentioned three-dimensional model editing methods.

[0014] In a fourth aspect, the present application also provides a computer program product, including a computer program, which implements the steps of any of the above-mentioned three-dimensional model editing methods when executed by a processor.

[0015] The present application provides a three-dimensional model editing method, device, storage medium and program product. First, multiple source images at different perspectives are obtained from a source three-dimensional model, and a target object in the source image is in a source state. Then, target text prompt information is determined. Since the target text prompt information includes target state information of the target object, the target state information refers to the state information of the target object in a target multi-perspective image obtained by editing based on the source image. Furthermore, the multiple source images and the target text prompt information are input into a multi-perspective image editing model that has been trained to convergence, so that the target multi-perspective images corresponding to the multiple source images can be output at one time. In the present application, the target multi-view image forms an image of the target state from the source state after the target object performs the target action under the perspective of the corresponding source image, thereby completing non-rigid editing. It can be seen that the present application implements editing of the target object state based on the target text prompt information, and can change the target object from one state to another, that is, from the source state to the target state; in addition, the multi-view image editing model that has been trained to convergence in the present application can input multiple source images at one time for simultaneous processing, and simultaneously output the target multi-view images corresponding to the multiple source images, thereby improving the editing speed; the present application uses source images under multiple perspectives to achieve editing, thereby combining multiple perspectives, so that the obtained three-dimensional model improves multi-perspective consistency and is more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0017] Figure 1 A schematic diagram of an application scenario of a three-dimensional model editing method provided in this application;

[0018] Figure 2A schematic diagram of a three-dimensional model editing method provided in Example 1;

[0019] Figure 3 A schematic diagram of a three-dimensional model editing method provided in Example 2;

[0020] Figure 4 A schematic diagram of a three-dimensional model editing method provided in Example 3;

[0021] Figure 5 A schematic diagram of a three-dimensional model editing method provided in Example 4;

[0022] Figure 6 A schematic diagram of a three-dimensional model editing method provided in Example 6;

[0023] Figure 7 A schematic diagram of a three-dimensional model editing method provided in Example 8;

[0024] Figure 8 A schematic diagram of a three-dimensional model editing method provided in Example 10;

[0025] Fig. 9 A schematic diagram of a three-dimensional model editing method provided in Example 11;

[0026] Fig.10 A schematic diagram of the overall flow of a three-dimensional model editing method provided in Example 13;

[0027] Fig.11 A schematic diagram of the structure of a three-dimensional model editing device provided in Example 14;

[0028] Fig.12 This is a schematic diagram of the structure of the editing device provided in Example 14. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0030] It should be noted that, in the description of this application, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. The terms "first", "second", etc. in this application are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0031] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0032] In conjunction with the specific application environment architecture or the specific hardware architecture on which the execution of the three-dimensional model editing method depends, the specific application environment architecture or the specific hardware architecture is described herein.

[0033] Low-Rank Adaptation (LoRA): By decomposing the weight matrix into the product of low-rank matrices, the number of parameters is reduced, thereby reducing hardware resources and accelerating fine-tuning and training processes.

[0034] In the related art, a pre-trained image editing model is used to edit the text description of the source image corresponding to the source 3D model to obtain the edited image, and then the edited image is promoted to a 3D mode by reconstruction to obtain the target 3D model, thereby achieving 3D editing. However, the above-mentioned related art can change some features of objects in the source 3D model in the same state or replace objects of the same type, but cannot process the state change of the target object.

[0035] In order to solve the defects of the related art, the inventor of the present invention has designed a new solution after creative research. The present invention provides a three-dimensional model editing method. In order to solve the problem that only partial feature changes or replacement of the same type of objects can be performed on the target object in the source image in the related art, the present application proposes to obtain multiple source images under different perspectives from the source three-dimensional model, the source image is in the source state, and then a multi-perspective image editing model that has been trained to convergence is used to output a plurality of target multi-perspective images corresponding to the source images. In the target multi-perspective image, it is reflected that the target object forms a target state from the source state after performing the target action. It can be seen that the target object changes from the source state to the target state, thereby realizing the change of state and realizing the non-rigid editing of the source image; in addition, the multi-perspective image editing model that has been trained to convergence in the present application is convergent, so that the target multi-perspective image output based on it is more accurate; in addition, the multi-perspective image editing model that has been trained to convergence simultaneously inputs multiple source images and simultaneously outputs multiple target multi-perspective images of the source objects after being edited, thereby improving the editing speed. In this application, non-rigid editing is performed on source images at different perspectives, which improves multi-perspective consistency and makes the target three-dimensional model reconstructed based on multiple target multi-perspective images more accurate.

[0036] Figure 1 A schematic diagram of an application scenario of a 3D model editing method provided in this application. Figure 1 As shown, an editing device 101 is included.

[0037] The editing device 101 may be any electronic device, such as a computer or a server, etc., and is not limited here.

[0038] In this scenario, the editing device 101 obtains a plurality of source images at different viewing angles from the source three-dimensional model. Further, the editing device 101 determines target text prompt information.

[0039] Furthermore, the editing device 101 inputs the source image and the target text prompt information into a multi-view image editing model that has been trained to convergence, and then outputs target multi-view images of the multiple source images at corresponding viewpoints.

[0040] Furthermore, the editing device 101 reconstructs a target three-dimensional model based on the target multi-view images.

[0041] The embodiment of the present application provides a three-dimensional model editing method, and the method is described in detail in conjunction with the execution flow of the three-dimensional model editing method.

[0042] Embodiment 1 The execution body of Embodiment 1 to Embodiment 13 of the present application is a three-dimensional model editing device, and the three-dimensional model editing device is located in the editing equipment.

[0043] Figure 2A schematic diagram of a three-dimensional model editing method provided in Example 1. Figure 2 As shown, specifically including:

[0044] S201, obtaining multiple source images at different viewing angles from a source three-dimensional model, where the source images are images corresponding to the three-dimensional model at different viewing angles; each source image includes a target object; and each target object is in a source state.

[0045] The source three-dimensional model refers to a three-dimensional model of a target object to be edited.

[0046] The target object refers to an object included in the source 3D model or the source image, and can be any object, for example, the target object is a cat or a dog. It should be noted that the source 3D model is a spatial 3D structure.

[0047] The source state refers to the state of the target object before editing.

[0048] The source image refers to the image corresponding to the source three-dimensional model at different viewing angles. It should be noted that the source image is two-dimensional, that is, flat.

[0049] S202, determining target text prompt information corresponding to each target multi-view image to be generated; the target text prompt information includes target state information of the target object.

[0050] The target text prompt information refers to the information for performing text prompt processing on the source image.

[0051] The target state refers to the state of the target object after editing.

[0052] Exemplarily, the target text prompt information includes a sitting yellow kitten, and the target text prompt information is information such as "sitting", "yellow" and "small", and the target state information is "sitting".

[0053] It should be noted that the target text prompt information also includes viewing angle information, for example, the target text prompt information may also include the viewing angle corresponding to the source image. Exemplarily, the target text prompt information may be a sitting yellow kitten in the viewing angle corresponding to the source image.

[0054] S203, input multiple source images and target text prompt information into a multi-view image editing model that has been trained to convergence, and output a target multi-view image of the multiple source images at corresponding viewpoints to achieve non-rigid editing; the target multi-view image is an image of a target state formed from a source state after the target object performs a target action at the viewpoint of the corresponding source image.

[0055] It should be noted that, in this embodiment, multiple source images are input together into a multi-view image editing model that has been trained to convergence, and target multi-view images of the multiple source images at corresponding viewpoints are output simultaneously.

[0056] Among them, the multi-view image editing model that has been trained to convergence refers to a model that combines multi-view editing of multiple source images based on target text prompt information, so that the target objects in multiple source images change from the source state to the target state under multi-view control.

[0057] In one approach, a multi-view image editing model that has been trained to convergence can receive multiple source images simultaneously, but process the multiple source images independently, and simultaneously output a target multi-view image corresponding to each source image.

[0058] In another way, the multi-view image editing model that has been trained to convergence can combine the view information between multiple source images, fully combine the multi-view consistency, and realize the non-rigid editing of multi-view consistency, so that the target object changes from the source state to the target state. It can be seen that each target multi-view image is obtained by combining the view between each source image.

[0059] It should be noted that, assuming there are four source images, the four source images are input into a multi-view image editing model that has been trained to convergence, and the multi-view image editing model that has been trained to convergence is used to simultaneously output the target multi-view images of the four source images at corresponding viewpoints, thereby obtaining four target multi-view images.

[0060] The perspective corresponding to the target multi-perspective image is the perspective corresponding to the source image, and the target multi-perspective image combines the perspectives corresponding to each source image, thereby achieving non-rigid editing with multi-perspective consistency.

[0061] The target object in the target multi-view image changes from a source state to a target state.

[0062] Among them, non-rigid editing refers to editing in which the state of the target object changes after executing the target action.

[0063] For example, assume that a source image is a front source image and a left side source image, the target object is a cat, the source state is "sitting", the target state is "standing", and the target object also has other original features, for example, the color is yellow, and the original features are not described here. In this example, the target text prompt information includes the target state information (that is, it means that the target object is changed from the source state to the target state, so the state of the target object changes), and also includes other features. In one way, the other features can be consistent with the original features in the source image, indicating that the original features are unchanged at this time.

[0064] Furthermore, the above-mentioned front source image, left side source image and target text prompt information are input into a multi-view image editing model that has been trained to convergence. The multi-view image editing model that has been trained to convergence combines the perspective information of the two source images and outputs a front target multi-view image corresponding to the front source image, and a left side target multi-view image corresponding to the left side source image. Among them, the front target multi-view image refers to an image in which the target object keeps the other original features unchanged and changes from the source state to the target state under the front perspective. Similarly, the left side target multi-view image refers to an image in which the target object keeps the other original features unchanged and changes from the source state to the target state under the left side perspective.

[0065] It should be noted that, in addition to the state change of the target object, the color and size of the target object in the source image can also be changed in the present application.

[0066] It should be noted that in the present application, the target state can be a local state change of the target object from the source state, or a global state change of the target object from the source state. It can be seen that the present application can realize global or local non-rigid editing of the target object; the editing device in the present application realizes global or local non-rigid editing of the source image, that is, it can realize non-rigid editing of global or local state change of the target object in the source image. It should be noted that the local state change refers to the state change of some parts of the target object. For example, the target object in the source image is a sitting cat with both hands vertically, and the target object in the target multi-view image in the source image is a sitting cat with the right hand raised and the left hand vertical. The global state change refers to the state change of all parts of the target object. For example, the target object is a sitting cat with both hands vertically, and the target object in the target multi-view image is a standing cat.

[0067] S204, reconstructing a target three-dimensional model based on the multi-view images of each target.

[0068] Specifically, an initial three-dimensional model is obtained by using a three-dimensional Gaussian representation according to each target multi-view image, and then a reconstruction loss function is applied to optimize the initial three-dimensional model to obtain a non-rigidly edited target three-dimensional model.

[0069] The present embodiment provides a three-dimensional model editing method. In the present embodiment, a plurality of source images under different perspectives are firstly acquired from a source three-dimensional model, and a target object in the source image is in a source state. Then, target text prompt information is determined. Since the target text prompt information includes target state information of the target object, the target state information refers to the state information of the target object in a target multi-perspective image obtained through editing based on the source image. Furthermore, the plurality of source images and the target text prompt information are input into a multi-perspective image editing model that has been trained to convergence, so that the target multi-perspective images corresponding to the plurality of source images can be output at one time. In this embodiment, the target multi-view image forms an image of the target state from the source state after the target object therein performs the target action under the perspective of the corresponding source image, so as to realize non-rigid editing. It can be seen that in this embodiment, the editing of the target object state is realized based on the target text prompt information, and the target object can be changed from one state to another, that is, from the source state to the target state; in addition, the multi-view image editing model that has been trained to convergence in this embodiment can input multiple source images at one time for simultaneous processing, and simultaneously output the target multi-view images corresponding to the multiple source images, so the editing speed is improved; in this embodiment, the source images under multiple perspectives are used to realize editing, thereby combining multiple perspectives, so that the obtained three-dimensional model is more accurate.

[0070] Embodiment 2

[0071] This embodiment is a further refinement of any of the above embodiments. This embodiment is an optional method of inputting multiple source images and target text prompt information into a multi-view image editing model that has been trained to convergence, and outputting target multi-view images of multiple source images at corresponding viewpoints to achieve non-rigid editing.

[0072] Figure 3 A schematic diagram of a three-dimensional model editing method provided in Example 2. Figure 3 As shown, specifically including:

[0073] S301, obtaining camera pose information of multiple source images.

[0074] The camera pose information refers to the position and orientation of the camera that captures the source image in space. It should be noted that the camera pose information can reflect the viewing angle of the source image.

[0075] S302, performing feature extraction on a plurality of source images at a preset time step to obtain a source image feature set at the preset time step.

[0076] The preset time step refers to a time step pre-set for the multi-view image editing model that has been trained to convergence.

[0077] The source image feature set refers to a feature set for preliminary feature extraction of the source image. The source image feature set includes the original features of multiple source images, that is, source image features. It can be seen that the source image feature set includes multiple source image features.

[0078] It should be noted that, in this step, the source image feature set can be expressed as follows (1).

[0079] (1)

[0080] in, , N represents the number of images in the source image feature set, is the size of the source image, C represents the feature dimension of the source image, t is the preset time step, is the source image feature set, is the source image feature of the i-th source image.

[0081] S303, obtaining a three-dimensional cost viewing frustum at a preset time step based on the position and posture information of each camera and the source image feature set.

[0082] The three-dimensional cost frustum is used to represent the range of a target object visible in a three-dimensional space and is a perspective model.

[0083] Among them, the three-dimensional cost frustum can represent the frustum after combining source images at different perspectives with multi-perspective consistency.

[0084] Specifically, the source image features corresponding to each source image are converted or distorted according to the camera pose information, and finally fused to obtain a three-dimensional cost frustum corresponding to a preset time step, that is, a 3D cost volume.

[0085] S304, using a multi-view image editing model trained to convergence to output a target multi-view image at corresponding viewpoints of multiple source images based on a three-dimensional cost frustum and target text prompt information, so as to achieve non-rigid editing.

[0086] This embodiment provides a three-dimensional model editing method. In this embodiment, camera posture information is first obtained, and then feature extraction is performed on multiple source images to obtain a source image feature set corresponding to a preset time step. Further, a three-dimensional cost frustum is obtained. Since the three-dimensional cost frustum can characterize the multi-view consistency of multiple source images, the target multi-view image output based on the three-dimensional cost frustum is more accurate.

[0087] Embodiment 3

[0088] This embodiment is a further refinement of any of the above embodiments. This embodiment is an optional way to obtain a three-dimensional cost viewing frustum at a preset time step based on the pose information of each camera and a source image feature set.

[0089] Figure 4 A schematic diagram of a three-dimensional model editing method provided in Example 3. Figure 4 As shown, including:

[0090] S401, determining each camera pose conversion matrix based on each camera pose information and preset conversion perspective information; the camera pose conversion matrix is ​​a conversion matrix that converts the perspective corresponding to the camera pose information to the preset conversion perspective information at the source depth; the camera pose information includes the perspective information corresponding to the source image.

[0091] The preset conversion perspective information refers to the information preset for converting from the corresponding perspective to the reference perspective. Exemplarily, assuming that there are four source images, namely, a source image of the front side, a source image of the left side, a source image of the right side, and a source image of the back side, the preset conversion perspective information may be the first perspective in the perspective image set. The perspective image set includes multiple source images. The first perspective may be the front side.

[0092] It should be noted that, assuming that the camera pose information of the source image on the left side reflects the perspective of the left side, then determining the camera pose transformation matrix corresponding to the source image on the left side is to determine the transformation matrix from the left side perspective to the reference perspective (first perspective) at the source depth.

[0093] It should be noted that the camera pose information includes the viewing angle information corresponding to the source image, so that the viewing angle corresponding to the source image can be determined from the viewing angle information in the camera pose information.

[0094] S402, inputting the camera pose transformation matrix and the source image feature set into a preset perspective transformation algorithm to output a transformation feature set of each source image transformed from the corresponding perspective to the preset transformation perspective at a preset time step.

[0095] The preset viewing angle conversion algorithm refers to a preset algorithm for converting the viewing angle of a source image.

[0096] The conversion feature set refers to the set of features of each source image converted from the corresponding viewing angle to the preset conversion viewing angle.

[0097] Exemplarily, the preset perspective conversion algorithm is shown as (2):

[0098] (2)

[0099] in, For the preset time step t, the source depth The distortion transformation diagram at Represents a pixel position in the reference view. At source depth The camera pose transformation matrix from the corresponding view to the reference view.

[0100] in, represents the conversion feature, and each conversion feature is combined into a conversion feature set. Wherein, D represents the source depth range.

[0101] S403, calculating the variance between each conversion feature set to obtain a three-dimensional cost frustum corresponding to a preset time step

[0102] Exemplarily, the variance calculation formula between each conversion feature set in this step is shown in (3):

[0103] (3)

[0104] Among them, Var represents the variance calculation. Among them, .

[0105] The present embodiment provides a three-dimensional model editing method. In the present embodiment, the camera posture transformation matrix is ​​first determined, and then input into a preset transformation perspective algorithm to calculate a transformation feature set. It can be seen that the transformation feature set combines different perspectives, thereby fusing multi-perspective features, thereby improving multi-perspective consistency; in addition, the three-dimensional cost frustum includes a source depth range, a source image size, and a source image feature dimension, so that the three-dimensional cost frustum is fully complete and more accurate.

[0106] Embodiment 4

[0107] This embodiment is a further refinement of any of the above embodiments. This embodiment uses a multi-view image editing model trained to convergence to output target multi-view images under corresponding viewpoints of multiple source images based on a three-dimensional cost frustum and target text prompt information to achieve an optional method of non-rigid editing.

[0108] Figure 5 A schematic diagram of a three-dimensional model editing method provided in Example 4. Figure 5 As shown, including:

[0109] S501, calculating a deep fusion feature at a preset time step based on three-dimensional cost view cone reshaping; the deep fusion feature is a fusion feature at a source depth.

[0110] Exemplarily, the deep fusion features are calculated based on the preset reshaping algorithm of formula (4).

[0111] (4)

[0112] in, represents the deep fusion feature, .

[0113] It should be noted that the deep fusion feature fuses features with the same depth.

[0114] S502, using a linear transformation layer to calculate a query vector, a key vector, and a value vector based on the deep fusion feature.

[0115] For example, the calculation method is shown in (5):

[0116] (5)

[0117] in, They are respectively the query vector linear transformation layer, the key vector linear transformation layer and the value vector linear transformation layer. .

[0118] S503, input the query vector, key vector and value vector into a preset three-dimensional self-attention mechanism algorithm to output a multi-view fusion feature at a preset time step; the multi-view fusion feature is a feature after fusion of multiple perspectives through the three-dimensional self-attention mechanism.

[0119] Among them, the preset three-dimensional self-attention mechanism algorithm is a preset three-dimensional self-attention mechanism algorithm, as shown in (6):

[0120] (6)

[0121] in, is a three-dimensional self-attention mechanism, is the multi-view fusion feature, where T represents transposition.

[0122] It should be noted that in this embodiment, multi-view self-attention calculation for depth perception is performed on the three-dimensional cost cone, that is, multi-view self-attention calculation is performed at the same source depth. Specifically, the principle is: different pixels of the same image will perform a self-attention mechanism with other images at the same depth, and a new multi-view image is obtained by fusing features from other views.

[0123] It should be noted that in this embodiment, the deep fusion feature is first obtained, and then the query vector, key vector and value vector are calculated respectively by using the linear transformation layer, and the multi-view features are further fused using the self-attention mechanism to obtain the multi-view fusion feature, thereby improving the multi-view consistency.

[0124] S504, using the multi-view image editing model that has been trained to convergence to output target multi-view images under the corresponding viewpoints of multiple source images based on multi-view fusion features and target text prompt information, so as to achieve non-rigid editing.

[0125] Furthermore, multi-view fusion features and target text prompt information are used to output target multi-view images under the corresponding viewpoints of multiple source images.

[0126] The present embodiment provides a three-dimensional model editing method. In the present embodiment, multi-perspective self-attention calculation of depth perception is performed on the basis of a three-dimensional cost frustum, thereby obtaining multi-perspective fusion features. In the present embodiment, the multi-perspective fusion features are fused from a query vector, a key vector, and a value vector, thereby realizing multi-perspective fusion and improving multi-perspective consistency.

[0127] Embodiment 5

[0128] This embodiment is a further refinement of any of the above embodiments. This embodiment uses a multi-view image editing model that has been trained to convergence to output a target multi-view image under the corresponding viewpoints of multiple source images based on multi-view fusion features and target text prompt information to achieve an optional method of non-rigid editing, including:

[0129] Step 1: According to the camera pose information corresponding to each source image, the multi-view fusion features are projected to the view corresponding to each source image to obtain a multi-view image feature set with depth perception at a preset time step; the multi-view image feature set is a feature set of multiple source images calculated by a three-dimensional self-attention mechanism with depth perception; the multi-view image feature set includes multi-view image features corresponding to multiple source images.

[0130] The multi-view image feature set is calculated by a depth-aware three-dimensional self-attention mechanism. In this embodiment, for the view corresponding to the source image, the editing device projects the multi-view fusion features to each view, thereby obtaining the multi-view image features of each view at a preset time step, and multiple multi-view image features constitute a multi-view image feature set.

[0131] Exemplarily, assuming that there are four source images at different viewing angles, namely the front, left, right and back, the editing device projects the multi-view fusion feature to the front to obtain the multi-view image feature of the front. Similarly, the multi-view fusion feature is projected to the left, right and back to obtain the multi-view image features at the corresponding viewing angles. It can be understood that in this example, four multi-view image features are obtained, and the above four multi-view image features constitute a multi-view image feature set.

[0132] Step 2: Use the multi-view image editing model that has been trained to convergence to output target multi-view images under the corresponding viewpoints of multiple source images based on the multi-view image feature set and the target text prompt information to achieve non-rigid editing.

[0133] The present embodiment provides a three-dimensional model editing method. In the present embodiment, a multi-perspective fusion feature is projected to the perspective corresponding to each source image, so that a multi-perspective image feature set with depth perception at a preset time can be obtained. Since the multi-perspective fusion feature improves the multi-perspective consistency, the multi-perspective image feature set also fuses multi-perspective information, thereby making the target multi-perspective image output by the multi-perspective image editing model that has been trained to convergence more accurate.

[0134] Embodiment 6

[0135] This embodiment is a further refinement of any of the above embodiments. This embodiment uses a multi-view image editing model that has been trained to convergence to output target multi-view images under the corresponding viewpoints of multiple source images based on a multi-view image feature set and target text prompt information, so as to achieve an optional method of non-rigid editing.

[0136] Figure 6 A schematic diagram of a three-dimensional model editing method provided in Example 6. Figure 6 As shown, specifically including:

[0137] S601, obtaining corresponding viewing angle information from the camera pose information of each source image.

[0138] S602, using a preset multi-layer perceptron to encode each viewing angle information to obtain each viewing angle code.

[0139] Among them, the preset multi-layer perceptron can encode the perspective information.

[0140] S603, using a multi-view image editing model that has been trained to convergence to output target multi-view images under the corresponding viewpoints of multiple source images based on the view encoding, the multi-view image feature set and the target text prompt information, so as to achieve non-rigid editing.

[0141] Specifically, a denoising condition is determined based on the view coding and the target text prompt information, and the multi-view image feature set is controlled to be edited and denoised based on the target text prompt information and the denoising condition, so as to obtain a target multi-view image.

[0142] The present embodiment provides a three-dimensional model editing method. In the present embodiment, the perspective information is first obtained based on the camera posture information, and then encoded to obtain the perspective code. Furthermore, the target multi-perspective image is output based on the perspective code, the multi-perspective image feature set and the target text prompt information. Since the multi-perspective image feature set integrates the multi-perspective features, the target multi-perspective image is more accurate and the multi-perspective consistency is improved.

[0143] Embodiment 7

[0144] This embodiment is a further refinement of any of the above embodiments. This embodiment uses a multi-view image editing model that has been trained to convergence to output target multi-view images at corresponding viewpoints of multiple source images based on each view encoding, multi-view image feature set, and target text prompt information to achieve an optional method of non-rigid editing, specifically including:

[0145] Step 1: Encode and optimize the target text prompt information to obtain optimized target text embedding data.

[0146] Specifically, a preset text editor can be used to encode the target text prompt information to obtain target text embedding data, and then the target text embedding data is optimized, and semantic compression is performed during the optimization process. Specifically: the semantic commonality vector and the semantic difference vector are decomposed based on the source text embedding data corresponding to the source image, and the optimized target text embedding data is calculated based on the semantic commonality vector, the semantic difference vector and the target text embedding data to obtain the optimized target text embedding data combining semantic commonality and semantic difference.

[0147] Step 2: Based on the multi-view image feature set, the encoding of each view and the optimized target text embedding data, the target multi-view image under the corresponding view of multiple source images is obtained to achieve non-rigid editing.

[0148] It should be noted that the optimized target text embedding data in this embodiment combines semantic commonalities and semantic differences to accurately control the degree of non-rigid editing of the source image, thereby making the obtained target multi-view image more accurate.

[0149] In one method, a multi-view image editing model that has been trained to convergence obtains a multi-view image feature in a multi-view image feature set, where the multi-view image feature corresponds to a source image Is1, and then outputs a target multi-view image corresponding to the source image Is1 based on the view encoding corresponding to the source image Is1 and the optimized target text embedding data; at the same time, for the remaining source images, the same method is used to output the corresponding target multi-view images.

[0150] This embodiment provides a three-dimensional model editing method. In this embodiment, the optimized target text embedding data can more accurately control non-rigid editing, so that the target multi-view image finally obtained is more accurate.

[0151] Embodiment 8

[0152] The present embodiment is a further refinement of any of the above embodiments. The present embodiment is an optional method for achieving non-rigid editing by obtaining target multi-view images at corresponding viewpoints of multiple source images based on a multi-view image feature set, encoding of each viewpoint and optimized target text embedding data.

[0153] Figure 7 A schematic diagram of a three-dimensional model editing method provided in Example 8. Figure 7 As shown, including:

[0154] S701, obtaining a preset time step.

[0155] S702, input each perspective encoding, preset time step, optimized target text embedding data and multiple source images into a preset convolution module to output denoising conditions of the multiple source images.

[0156] Among them, the preset convolution module can be a ResBlock basic module.

[0157] It should be noted that in this embodiment, each perspective encoding, preset time step, optimized target text embedding data and multiple source images are input into the preset convolution module as conditional information, so that the denoising conditions of multiple source images can be output.

[0158] S703, controlling the multi-view image feature set to perform a target action according to the optimized target text embedding data to obtain a multi-view image edited at a view corresponding to the view encoding.

[0159] It should be noted that the multi-view image feature set includes multi-view image features corresponding to multiple source images. For each multi-view image feature, the target action is performed according to the corresponding optimized target text embedding data, so as to obtain a multi-view image at the perspective corresponding to the perspective encoding.

[0160] The multi-view image refers to an edited, non-denoised image that combines multiple viewpoints.

[0161] The denoising condition refers to the condition under which denoising is required during the editing process.

[0162] It should be noted that, in this embodiment, when the multi-view image is obtained through the cross-attention mechanism, the multi-view image editing model that has been trained to convergence can dynamically adjust the image features according to the relationship between each part of the source image and the conditional information, so that the multi-view image editing model that has been trained to convergence controls the generation of relevant parts of the multi-view image according to the perspective encoding and the optimized target text embedding data, so as to realize that the target object performs the target action in the source state to form the target state, and generates a multi-view image at the perspective corresponding to the perspective encoding.

[0163] S704, denoising each edited multi-view image based on a denoising condition to obtain a target multi-view image at a corresponding view to achieve non-rigid editing.

[0164] Among them, the denoising conditions can include perspective encoding and preset time steps, so that the time step and perspective encoding can be controlled during the editing process, the noise level can be controlled, and denoising can be completed, thereby realizing perspective control design. Driven by the optimized target text embedding data, the multi-perspective image editing model that has been trained to convergence can effectively combine temporal dynamics, perspective information and text prompts to generate high-quality and perspective-controllable images.

[0165] The present embodiment provides a three-dimensional model editing method. In the present embodiment, denoising conditions are obtained, and then denoising is performed based on the denoising conditions on the basis of the generated multi-view image, so that the obtained target multi-view image has higher quality; and in the present embodiment, the multi-view image editing model that has been trained to convergence can effectively combine temporal dynamics, view information and optimized target text embedding data.

[0166] Embodiment 9

[0167] The present embodiment is a further refinement of any of the above embodiments. In the present embodiment, the multi-view image editing model that has been trained to convergence is a model in which a three-dimensional self-attention module is added to a diffusion model that has been trained to convergence.

[0168] The computational layers of query vector, key vector and value vector in the 3D self-attention module are added with low-rank adaptive layers respectively.

[0169] The diffusion model that has been trained to convergence may be a pre-trained diffusion model.

[0170] It should be noted that the three-dimensional self-attention module in this embodiment can process the source image from multiple perspectives, so that the obtained target multi-perspective image is combined with multiple perspectives to improve the consistency of multiple perspectives.

[0171] The present embodiment provides a three-dimensional model editing method. In the present embodiment, since the multi-view image editing model that has been trained to convergence is a model in which a three-dimensional self-attention module is added to a diffusion model that has been trained to convergence, due to the addition of the three-dimensional self-attention module, the multi-view image editing model that has been trained to convergence can better combine multiple perspectives, and calculate the input source images of different perspectives through the three-dimensional self-attention mechanism, thereby fusing different perspective information, outputting a valid target multi-view image, and outputting multi-view consistency.

[0172] Embodiment 10

[0173] This embodiment is a further refinement of any of the above embodiments. This embodiment is an optional method before inputting multiple source images and target text prompt information into a multi-view image editing model that has been trained to convergence, and outputting target multi-view images of multiple source images at corresponding viewpoints to achieve non-rigid editing.

[0174] Figure 8 A schematic diagram of a three-dimensional model editing method provided in Example 10. Figure 8 As shown, including:

[0175] S801, obtaining a training data set; the training data set includes at least one source training image under different viewing angles and its corresponding edited image.

[0176] The training data set can be obtained from an open source database, from which at least one image under different viewing angles is determined as a source training image.

[0177] In one approach, at least one source training image may be input into a diffusion model that has been trained to convergence to obtain a corresponding edited image.

[0178] S802, keeping the diffusion model that has been trained to convergence unchanged, and training the training parameters of the low-rank adaptive layer until the convergence condition is reached.

[0179] The convergence condition refers to the condition under which the multi-view image editing model reaches convergence. The convergence condition may include the number of training times or whether the training result reaches a preset consistency with the image edited in step S801, or whether the output multi-view diffusion loss value meets a preset loss value.

[0180] Exemplarily, in this step, when training the multi-view image editing model, a multi-view diffusion loss function is used, as shown in (7):

[0181] (7)

[0182] in, shows the true value of noise at N viewing angles, is the input image with noise from N perspectives, that is, the image formed by adding noise to the edited image, represents the parameter weights of the multi-view image editing model, represents the optimized training text embedding data, represents the preset time step of the multi-view image editing model, represents the view encoding of N views, represents the image to be edited, (1:N) represents from the first perspective to the Nth perspective, represents the multi-view diffusion loss value, Refers to the multi-view image editing model.

[0183] Specifically, the training data set is input into the multi-view image editing model, and the multi-view diffusion loss value corresponding to the training data set is obtained by using the above (7). The training parameters of the low-rank adaptive layer in the multi-view image editing model are trained based on the multi-view diffusion loss value until the convergence condition is reached.

[0184] S803: Determine the multi-view image editing model that has reached the convergence condition as the multi-view image editing model that has been trained to convergence.

[0185] This embodiment provides a three-dimensional model editing method. During training in this embodiment, since the diffusion model that has been trained to convergence has been trained, only the training parameters of the low-rank adaptive layer are changed for training, and the diffusion model that has been trained to convergence is kept unchanged, thereby accelerating network convergence and accelerating the training process.

[0186] Embodiment 11

[0187] This embodiment is a further refinement of any of the above embodiments, and this embodiment is an optional way to obtain a training data set.

[0188] Fig. 9 A schematic diagram of a three-dimensional model editing method provided in Example 11. Fig. 9 As shown, specifically including:

[0189] S901, obtaining a preset 3D object dataset; the preset 3D object dataset includes at least one preset 3D object.

[0190] The preset 3D object dataset may be an open source dataset or a user private dataset.

[0191] The preset 3D object data set further includes a text description of at least one preset 3D object.

[0192] S902: For each preset 3D object, obtain at least one image to be edited corresponding to a preset elevation angle and a preset training viewing angle.

[0193] The preset elevation angle refers to a preset elevation angle when facing the center of the preset 3D object. For example, the preset elevation angle may be between 0 and 30 degrees.

[0194] The preset training perspective refers to a preset perspective facing a preset 3D object. It should be noted that the preset training perspective can be the front, left, right, and back, or other perspectives, which are not limited here.

[0195] Exemplarily, assuming that M preset 3D objects are obtained, at least one image to be edited corresponding to the preset elevation angle and the preset training perspective is obtained for each preset 3D object. Specifically, assuming that the preset elevation angle is 10°, and the preset training perspectives are the front, left, right, and back, then for one preset 3D object, when the preset elevation angle is 10°, an image to be edited of the front side is obtained when the preset training perspective is the preset elevation angle, when the preset elevation angle is 10°, an image to be edited of the left side is obtained when the preset training perspective is the preset elevation angle, when the preset elevation angle is 10°, an image to be edited of the right side is obtained when the preset elevation angle is 10°, and when the preset training perspective is the preset elevation angle, an image to be edited of the back side is obtained when the preset training perspective is the preset elevation angle, and thus four images to be edited can be obtained.

[0196] Exemplarily, for M preset 3D objects, 4M images to be edited may be obtained.

[0197] S903: Acquire a training data set based on at least one image to be edited.

[0198] In one embodiment, the 4M images to be edited obtained in S902 are input into any image editing model that has been trained to convergence or a single-view image editing model that has been trained to convergence, so as to output corresponding edited images. It is understood that the number of edited images is 4M.

[0199] Among them, the single-view image editing model that has been trained to convergence refers to a model that performs single-view editing on a source image based on target text prompt information, so that the target object in the source image changes from a source state to a target state under single-view control.

[0200] The image editing model that has been trained to convergence can be any diffusion model used to edit images.

[0201] Furthermore, a training data set is determined from the 4M images to be edited and the corresponding 4M edited images.

[0202] This embodiment provides a three-dimensional model editing method. In this embodiment, at least one image to be edited is specifically determined from at least one preset 3D object according to a preset elevation angle and a preset training viewing angle, and then a training data set is acquired based on the image to be edited.

[0203] Embodiment 12

[0204] This embodiment is a further refinement of any of the above embodiments. This embodiment is an optional method for obtaining a training data set based on at least one image to be edited, including:

[0205] Step 1: Input at least one image to be edited into a single-view image editing model that has been trained to convergence to output corresponding edited image subsets; the edited image subsets include edited images corresponding to at least one image to be edited.

[0206] The single-view image editing model trained to convergence is a model in which a low-rank adaptive layer is added to the key vector and value vector of the cross-attention module of the single-view image generation model trained to convergence. The single-view image generation model trained to convergence is a model obtained by continuing to train on the basis of the basic diffusion model trained to convergence by optimizing the input and the included training parameters.

[0207] Exemplarily, assuming that 4M images to be edited are respectively input into a single-view image editing model that has been trained to convergence, and the corresponding edited images are output respectively, and finally 4M edited images are obtained, and the 4M edited images are determined as the edited image subset.

[0208] Step 2: Continue to input at least one edited image into the single-view image editing model that has been trained to convergence to output a new edited image subset, until the obtained edited image set meets the data preparation conditions, and determine at least one source training image under different perspectives and its corresponding edited image from at least one image to be edited and the edited image set to obtain a training data set; the edited image set is composed of at least one edited image subset; the source training image is determined from the image to be edited.

[0209] Furthermore, in order to ensure accuracy and multi-view consistency, at least one edited image is input into a single-view image editing model that has been trained to convergence, so as to output new edited images and obtain a new edited image subset, and this process is repeated until the data preparation conditions are met.

[0210] The data preparation condition refers to the condition for preparing a training data set. For example, the edited image set includes a preset number of edited images, thereby satisfying the data preparation condition. For example, the data preparation condition may be a condition that the edited images are greater than or equal to 64M. If the edited image set includes 64M edited images, it is determined that the edited image set meets the data preparation condition.

[0211] It should be noted that the edited image set includes at least one edited image (i.e., edited image subset) output by the single-view image editing model that has been trained to convergence each time. Exemplarily, assuming that after 16 cycles, 16 edited image subsets are obtained, and then the edited image set consists of the 16 edited image subsets.

[0212] Exemplarily, there are 4M images to be edited, and at least one image to be edited is selected from the 4M images to be edited as a source training image, wherein the source training image includes images under different viewing angles. An edited image corresponding to the source training image is selected from the edited image set, and the training object in the edited image can more accurately represent the change from the initial state of training to the final state of training after the training action.

[0213] It should be noted that the single-view image editing model that has been trained to convergence only inputs one image to be edited each time.

[0214] The present embodiment provides a three-dimensional model editing method. In the present embodiment, at least one image to be edited is input into a single-view image editing model that has been trained to convergence to obtain at least one corresponding edited image. Then, the at least one edited image obtained each time is repeatedly input into the single-view image editing model that has been trained to convergence to obtain a new edited image subset. This process is repeated until the edited image set meets the data preparation conditions, and then at least one source training image is determined from the at least one image to be edited, and at least one edited image is determined from the edited image set. The training data set finally obtained has improved multi-view consistency and is more accurate.

[0215] Embodiment 13

[0216] Fig.10 This is a schematic diagram of the overall flow of a three-dimensional model editing method provided in Example 13. Fig.10 As shown, it includes a source three-dimensional model 1001, multiple source images 1002, a multi-view image editing model 1003 that has been trained to convergence, a target multi-view image 1004 and a target three-dimensional model 1005.

[0217] Among them, in the multi-view image editing model 1003 that has been trained to convergence, a three-dimensional cost view frustum is first obtained, and then a multi-view image feature set is obtained based on the three-dimensional cost view frustum through a multi-view self-attention mechanism with depth perception, and the target multi-view image 1004 corresponding to the multiple source images 1002 is obtained after a cross-attention mechanism, view encoding and optimized target text embedding data.

[0218] Fig.10 In the example, two source images 1002 are used for illustration, wherein one source image 1002 is a front source image 1002, and the other source image 1002 is a left side source image 1002. Fig.10 Finally, the target multi-view image 1004 corresponding to the two source images is output simultaneously.

[0219] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method.

[0220] Embodiment 14

[0221] The following is an embodiment of the device of the present application. Fig.11 This is a schematic diagram of the structure of a 3D model editing device provided in Example 14. The 3D model editing device 1100 includes the following modules:

[0222] The acquisition module 1101 is used to acquire multiple source images at different viewing angles from the source three-dimensional model, where the source images are images corresponding to the three-dimensional model at different viewing angles; each source image includes a target object; and each target object is in a source state;

[0223] A determination module 1102 is used to determine target text prompt information corresponding to each target multi-view image to be generated; the target text prompt information includes target state information of the target object;

[0224] The output module 1103 is used to input multiple source images and target text prompt information into the multi-view image editing model that has been trained to convergence, and output the target multi-view images of the multiple source images at corresponding viewpoints to achieve non-rigid editing; the target multi-view image is an image of the target state formed from the source state after the target object performs the target action at the viewpoint corresponding to the source image;

[0225] The reconstruction module 1104 is used to reconstruct the target three-dimensional model based on the multi-view images of each target.

[0226] Optionally, when the output module 1103 inputs a plurality of source images and target text prompt information into a multi-view image editing model that has been trained to convergence and outputs a plurality of target multi-view images of the source images at corresponding viewpoints to implement non-rigid editing, it is specifically used to:

[0227] Get the camera pose information of multiple source images;

[0228] Performing feature extraction on multiple source images at a preset time step to obtain a source image feature set at the preset time step;

[0229] Based on the pose information of each camera and the feature set of the source image, a three-dimensional cost frustum is obtained at a preset time step;

[0230] A multi-view image editing model trained to convergence is used to output target multi-view images under the corresponding viewpoints of multiple source images based on the three-dimensional cost frustum and target text prompt information to achieve non-rigid editing.

[0231] Optionally, when the output module 1103 obtains the three-dimensional cost view frustum at a preset time step based on the position information of each camera and the source image feature set, it is specifically used to:

[0232] Determine each camera pose conversion matrix based on each camera pose information and preset conversion view information; the camera pose conversion matrix is ​​a conversion matrix that converts the view corresponding to the camera pose information to the preset conversion view information at the source depth; the camera pose information includes the view information corresponding to the source image;

[0233] Input the camera pose transformation matrix and the source image feature set into a preset perspective conversion algorithm to output a transformation feature set of each source image when it is transformed from the corresponding perspective to the preset transformation perspective at a preset time step;

[0234] The variance between each transformed feature set is calculated to obtain the three-dimensional cost frustum corresponding to the preset time step.

[0235] Optionally, when the output module 1103 uses the multi-view image editing model trained to convergence to output target multi-view images at corresponding viewpoints of multiple source images based on the three-dimensional cost frustum and target text prompt information to achieve non-rigid editing, it is specifically used to:

[0236] The deep fusion feature at the preset time step is calculated based on the three-dimensional cost view cone reshaping; the deep fusion feature is the fusion feature at the source depth;

[0237] Based on the deep fusion features, a linear transformation layer is used to calculate the query vector, key vector, and value vector;

[0238] Input the query vector, key vector and value vector into a preset three-dimensional self-attention mechanism algorithm to output a multi-view fusion feature at a preset time step; the multi-view fusion feature is a feature after fusion of multiple views by the three-dimensional self-attention mechanism;

[0239] A multi-view image editing model that has been trained to convergence is used to output target multi-view images under the corresponding viewpoints of multiple source images based on multi-view fusion features and target text prompt information to achieve non-rigid editing.

[0240] Optionally, when the output module 1103 uses the multi-view image editing model that has been trained to convergence to output the target multi-view images under the corresponding viewpoints of multiple source images based on the multi-view fusion features and the target text prompt information to achieve non-rigid editing, it is specifically used to:

[0241] According to the camera pose information corresponding to each source image, the multi-view fusion feature is projected to the view corresponding to each source image to obtain a multi-view image feature set with depth perception at a preset time step; the multi-view image feature set is a feature set of multiple source images calculated by a three-dimensional self-attention mechanism with depth perception; the multi-view image feature set includes multi-view image features corresponding to multiple source images;

[0242] A multi-view image editing model that has been trained to convergence is used to output target multi-view images under the corresponding viewpoints of multiple source images based on the multi-view image feature set and target text prompt information to achieve non-rigid editing.

[0243] Optionally, when the output module 1103 uses the multi-view image editing model that has been trained to convergence to output the target multi-view images under the corresponding viewpoints of multiple source images based on the multi-view image feature set and the target text prompt information to achieve non-rigid editing, it is specifically used to:

[0244] Obtaining corresponding viewing angle information from the camera pose information of each source image;

[0245] A preset multi-layer perceptron is used to encode the information of each viewing angle to obtain the encoding of each viewing angle;

[0246] A multi-view image editing model that has been trained to convergence is used to output target multi-view images under the corresponding viewpoints of multiple source images based on the view encoding, multi-view image feature set and target text prompt information to achieve non-rigid editing.

[0247] Optionally, when the output module 1103 uses the multi-view image editing model that has been trained to convergence to output the target multi-view images under the corresponding view angles of multiple source images based on the view angle encoding, the multi-view image feature set and the target text prompt information to achieve non-rigid editing, it is specifically used to:

[0248] Encode and optimize the target text prompt information to obtain optimized target text embedding data;

[0249] Based on the multi-view image feature set, the encoding of each view and the optimized target text embedding data, the target multi-view image under the corresponding view of multiple source images is obtained to achieve non-rigid editing.

[0250] Optionally, when the output module 1103 obtains the target multi-view images at the corresponding viewpoints of the plurality of source images based on the multi-view image feature set, the encoding of each viewpoint and the optimized target text embedding data to realize non-rigid editing, it is specifically used to:

[0251] Get the preset time step;

[0252] Input each view encoding, preset time step, optimized target text embedding data and multiple source images into a preset convolution module to output denoising conditions of multiple source images;

[0253] Controlling the multi-view image feature set to perform a target action according to the optimized target text embedding data to obtain a multi-view image edited at a view corresponding to the view encoding;

[0254] De-noising is performed on each edited multi-view image based on a denoising condition to obtain a target multi-view image at a corresponding viewing angle.

[0255] Optionally, the multi-view image editing model that has been trained to convergence is a model in which a three-dimensional self-attention module is added to a diffusion model that has been trained to convergence;

[0256] The computational layers of query vector, key vector and value vector in the 3D self-attention module are added with low-rank adaptive layers respectively.

[0257] Optionally, before inputting the plurality of source images and the target text prompt information into the multi-view image editing model that has been trained to convergence and outputting the target multi-view images of the plurality of source images at corresponding viewpoints to achieve non-rigid editing, the present embodiment provides a three-dimensional model editing device, further comprising: a training module;

[0258] The acquisition module 1101 is further used to acquire a training data set; the training data set includes at least one source training image under different viewing angles and its corresponding edited image;

[0259] A training module is used to keep the diffusion model that has been trained to convergence unchanged and train the training parameters of the low-rank adaptive layer until the convergence condition is reached;

[0260] The determination module 1102 is further configured to determine the multi-view image editing model that has reached a convergence condition as a multi-view image editing model that has been trained to convergence.

[0261] Optionally, when acquiring the training data set, the acquisition module 1101 is specifically used to:

[0262] Acquire a preset 3D object data set; the preset 3D object data set includes at least one preset 3D object;

[0263] For each preset 3D object, obtaining at least one image to be edited corresponding to a preset elevation angle and a preset training viewing angle;

[0264] A training dataset is obtained based on at least one image to be edited.

[0265] Optionally, when acquiring the training data set based on at least one image to be edited, the acquisition module 1101 is specifically configured to:

[0266] Inputting at least one image to be edited into a single-view image editing model that has been trained to convergence, respectively, to output corresponding edited image subsets; the edited image subsets include edited images corresponding to at least one image to be edited;

[0267] Continue to input at least one edited image into the single-view image editing model that has been trained to convergence to output a new edited image subset, until the obtained edited image set meets the data preparation conditions, determine at least one source training image under different perspectives and its corresponding edited image from at least one image to be edited and the edited image set to obtain a training data set; the edited image set is composed of at least one edited image subset; the source training image is determined from the image to be edited.

[0268] The description of the features in the embodiment corresponding to the three-dimensional model editing device can refer to the relevant description of the embodiment corresponding to the three-dimensional model editing method, which will not be repeated here.

[0269] The embodiment of the present application also provides an editing device, Fig.12 This is a schematic diagram of the structure of the editing device provided in Example 14. Fig.12 As shown, the editing device 1200 includes a processor 1201 and a memory 1202, wherein the processor 1201, the memory 1202 and the communication component 1203 are connected via a bus 1204. The memory 1202 stores a computer program, and the processor 1201 is configured to run the computer program to execute the steps in any of the above three-dimensional model editing method embodiments.

[0270] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any one of the above-mentioned three-dimensional model editing method embodiments when running.

[0271] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0272] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any one of the above-mentioned three-dimensional model editing method embodiments are implemented.

[0273] An embodiment of the present application also provides another computer program product, including a non-volatile computer-readable storage medium, the non-volatile computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the steps in any of the above-mentioned three-dimensional model editing method embodiments are implemented.

[0274] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0275] The above is a detailed introduction to a three-dimensional model editing method provided by the present application. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and its core idea of ​​the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A three-dimensional model editing method, characterized in that: include: Acquire multiple source images at different viewing angles from a source three-dimensional model, wherein the source images are images corresponding to the three-dimensional model at different viewing angles; each source image includes a target object; and each target object is in a source state; Determine target text prompt information corresponding to each target multi-view image to be generated; The target text prompt information includes target state information of the target object; Inputting the plurality of source images and target text prompt information into a multi-view image editing model that has been trained to convergence, and outputting target multi-view images of the plurality of source images at corresponding viewpoints, so as to realize non-rigid editing; the target multi-view image is an image of a target state formed from the source state after the target object performs a target action at the viewpoint corresponding to the source image; The target 3D model is reconstructed based on the multi-view images of each target.

2. The method according to claim 1, characterized in that The step of inputting the plurality of source images and the target text prompt information into a multi-view image editing model that has been trained to convergence, and outputting target multi-view images of the plurality of source images at corresponding viewpoints to achieve non-rigid editing includes: Obtaining camera pose information of the multiple source images; Performing feature extraction on the plurality of source images at a preset time step to obtain a source image feature set at the preset time step; Obtaining a three-dimensional cost viewing frustum at the preset time step based on each of the camera pose information and the source image feature set; A multi-view image editing model trained to convergence is used to output target multi-view images at corresponding view angles of the multiple source images based on the three-dimensional cost frustum and target text prompt information to achieve non-rigid editing.

3. The method according to claim 2, characterized in that The obtaining of the three-dimensional cost frustum at the preset time step based on the camera pose information and the source image feature set includes: Determine each camera pose conversion matrix based on each of the camera pose information and the preset conversion view information; the camera pose conversion matrix is ​​a conversion matrix that converts the view corresponding to the camera pose information to the preset conversion view information at the source depth; the camera pose information includes the view information corresponding to the source image; Input the camera pose transformation matrix and the source image feature set into a preset perspective conversion algorithm to output a transformation feature set of each source image when it is transformed from the corresponding perspective to the preset transformation perspective at a preset time step; The variance between each conversion feature set is calculated to obtain the three-dimensional cost frustum corresponding to the preset time step.

4. The method according to claim 2, characterized in that: The multi-view image editing model trained to convergence outputs the target multi-view images at the corresponding view angles of the multiple source images based on the three-dimensional cost frustum and the target text prompt information to achieve non-rigid editing, including: Calculating a deep fusion feature at the preset time step based on the three-dimensional cost view cone reshaping; the deep fusion feature is a fusion feature at the source depth; Calculate the query vector, key vector and value vector using a linear transformation layer based on the deep fusion feature; Inputting the query vector, the key vector and the value vector into a preset three-dimensional self-attention mechanism algorithm to output a multi-view fusion feature at the preset time step; the multi-view fusion feature is a feature after fusion of multiple views by the three-dimensional self-attention mechanism; The multi-view image editing model that has been trained to convergence is used to output the target multi-view images under the corresponding viewpoints of the multiple source images based on the multi-view fusion features and the target text prompt information to achieve non-rigid editing.

5. The method according to claim 4, characterized in that The method of using the multi-view image editing model that has been trained to convergence to output the target multi-view images at the corresponding viewpoints of the multiple source images based on the multi-view fusion features and the target text prompt information to achieve non-rigid editing includes: According to the camera pose information corresponding to each source image, the multi-view fusion feature is projected to the view corresponding to each source image to obtain a multi-view image feature set of depth perception at the preset time step; the multi-view image feature set is a feature set of the multiple source images calculated by the three-dimensional self-attention mechanism of depth perception; the multi-view image feature set includes multi-view image features corresponding to multiple source images; The multi-view image editing model that has been trained to convergence is used to output the target multi-view images under the corresponding viewpoints of the multiple source images based on the multi-view image feature set and the target text prompt information to achieve non-rigid editing.

6. The method according to claim 5, characterized in that The method of using the multi-view image editing model that has been trained to convergence to output the target multi-view images at the corresponding viewpoints of the plurality of source images based on the multi-view image feature set and the target text prompt information to achieve non-rigid editing includes: Obtain corresponding viewing angle information from the camera pose information of each source image; A preset multi-layer perceptron is used to encode the information of each viewing angle to obtain the encoding of each viewing angle; The multi-view image editing model that has been trained to convergence is used to output the target multi-view images under the corresponding view angles of the multiple source images based on the view encodings, the multi-view image feature set and the target text prompt information to achieve non-rigid editing.

7. The method according to claim 6, characterized in that The method of using the multi-view image editing model that has been trained to convergence to output the target multi-view images at the corresponding viewpoints of the multiple source images based on the viewpoint encodings, the multi-view image feature set, and the target text prompt information to achieve non-rigid editing includes: Encoding and optimizing the target text prompt information to obtain optimized target text embedding data; Based on the multi-view image feature set, each of the view encodings and the optimized target text embedding data, a target multi-view image under the corresponding view angles of the multiple source images is obtained to achieve non-rigid editing.

8. The method according to claim 7, characterized in that The method of obtaining the target multi-view images at the corresponding viewpoints of the plurality of source images based on the multi-view image feature set, each viewpoint encoding and the optimized target text embedding data to realize non-rigid editing includes: Get the preset time step; Inputting each of the perspective codes, the preset time step, the optimized target text embedding data and the multiple source images into a preset convolution module to output denoising conditions of the multiple source images; Controlling the multi-view image feature set to perform a target action according to the optimized target text embedding data to obtain a multi-view image edited at a view corresponding to the view encoding; De-noising is performed on each of the edited multi-view images based on the de-noising condition to obtain a target multi-view image at a corresponding viewing angle, so as to achieve non-rigid editing.

9. The method according to claim 1, characterized in that: The multi-view image editing model that has been trained to convergence is a model in which a three-dimensional self-attention module is added to a diffusion model that has been trained to convergence; The calculation layers of the query vector, key vector and value vector in the three-dimensional self-attention module are respectively added with low-rank adaptive layers.

10. The method according to claim 9, characterized in that Before inputting the plurality of source images and the target text prompt information into the multi-view image editing model that has been trained to convergence and outputting the target multi-view images of the plurality of source images at corresponding viewpoints to achieve non-rigid editing, the method further includes: Acquire a training data set; the training data set includes at least one source training image under different viewing angles and its corresponding edited image; Keeping the diffusion model that has been trained to convergence unchanged, training the training parameters of the low-rank adaptive layer until a convergence condition is reached; The multi-view image editing model that reaches the convergence condition is determined as the multi-view image editing model that has been trained to convergence.

11. The method according to claim 10, characterized in that The step of obtaining a training data set includes: Acquire a preset 3D object data set; the preset 3D object data set includes at least one preset 3D object; For each of the preset 3D objects, obtaining at least one image to be edited corresponding to a preset elevation angle and a preset training viewing angle; A training data set is acquired based on the at least one image to be edited.

12. The method according to claim 11, characterized in that The step of acquiring a training data set based on the at least one image to be edited comprises: Inputting the at least one image to be edited into a single-view image editing model that has been trained to convergence, respectively, to output corresponding edited image subsets, respectively; the edited image subsets include edited images corresponding to the at least one image to be edited; Continue to input at least one edited image into the single-view image editing model that has been trained to convergence to output a new edited image subset, until the obtained edited image set meets the data preparation conditions, determine at least one source training image under different perspectives and its corresponding edited image from the at least one image to be edited and the edited image set to obtain a training data set; the edited image set is composed of at least one edited image subset; the source training image is determined from the image to be edited.

13. An editing device, characterized in that: include: Memory and processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method according to any one of claims 1 to 12.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 12 when executed by a processor.

15. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 12 when being executed by a processor.

Images