Three-dimensional model editing method and device, storage medium and program product

Abstract

The invention discloses a three-dimensional model editing method and device, a storage medium and a program product, and relates to the technical field of image processing, a plurality of source images under different visual angles are obtained from a source three-dimensional model, and initial text prompt information and target text prompt information corresponding to each source image are determined; the target text prompt information comprises target state information of the target object; determining source text embedded data corresponding to each source image according to each piece of initial text prompt information; determining corresponding target text embedded data according to each target text prompt message, wherein each target single-view image is an image which forms a target state from a source state after the target object executes a target action under the view of the corresponding source image; and for each source image, determining a target single-view image corresponding to the source image based on the source text embedded data corresponding to the source image and the target text embedded data, so that the state of the target object is changed, and non-rigid editing is realized.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a three-dimensional model editing method, device, storage medium and program product. Background Art

[0002] Nowadays, three-dimensional dynamic changes are widely used in augmented / virtual reality, film / game production, and artistic creation, so how to achieve high-quality three-dimensional dynamic changes is particularly important.

[0003] In the related art, the editor can replace the objects in the source 3D model with the same type or change other minor features in the same state. However, the related art cannot handle when changing the objects in the source 3D model from one state to another. Summary of the invention

[0004] The present application provides a three-dimensional model editing method, device, storage medium and program product to achieve the effect of changing the state of a target object.

[0005] In a first aspect, the present application provides a three-dimensional model editing method, comprising:

[0006] A plurality of source images at different viewing angles are obtained from a source three-dimensional model, where the source images are images corresponding to the three-dimensional model at different viewing angles; each source image includes a target object; and the target object is in a source state;

[0007] Determine initial text prompt information corresponding to each source image and target text prompt information corresponding to each target single-view image to be generated, wherein the initial text prompt information includes source state information of the target object, and the target text prompt information includes target state information of the target object;

[0008] Determine the source text embedding data corresponding to each source image according to each initial text prompt information; and determine the target text embedding data corresponding to each target single-view image to be generated according to each target text prompt information, each target single-view image is an image formed from a source state to a target state after a target object performs a target action under the perspective of the corresponding source image;

[0009] For each source image, a single-view image editing model that has been trained to convergence is used to determine a target single-view image corresponding to the source image based on source text embedding data and target text embedding data corresponding to the source image, so as to achieve non-rigid editing;

[0010] The target 3D model is reconstructed based on the single-view images of each target.

[0011] In a second aspect, an embodiment of the present application provides an editing device, including: a memory and a processor;

[0012] Memory stores computer-executable instructions;

[0013] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the above first aspect and / or various possible implementations of the first aspect.

[0014] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the first aspect above and / or various possible implementation methods of the first aspect.

[0015] In a fourth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the above first aspect and / or various possible implementations of the first aspect.

[0016] The three-dimensional model editing method, device, storage medium and program product provided in the embodiments of the present application first obtain the corresponding source images under different perspectives from the source three-dimensional model, then determine the initial text prompt information and target text prompt information corresponding to each source image, wherein the initial text prompt information includes the source state information of the target object, and the target text prompt information includes the target state information of the target object, then determine the source text embedding data of the corresponding source image according to each initial text prompt information, further, determine the target text embedding data according to the target text prompt information, and use it to generate a target single-perspective image, wherein the target single-perspective image is an image that forms a target state from the source state after the target object performs a target action under the perspective of the corresponding source image, further, according to each source image, the corresponding target single-perspective image is determined respectively using a single-perspective image editing model that has been trained to convergence, so that the edited target single-perspective image corresponding to each source image can be obtained, thereby editing and reconstructing the target three-dimensional model based on each target single-perspective image. It can be seen that in this application, with the help of target text prompt information, the source image can perform the target action in the corresponding source state to form a target single-view image of the target state; in addition, the present application can realize non-rigid editing of changes in the state of the target object under the action of the target text prompt information, and then realize text-driven non-rigid editing of the image; in this application, for the source three-dimensional model, the source images are obtained from multiple perspectives, and the edited target single-view images are obtained from multiple source images. Since the present application considers multiple perspectives, the multi-perspective consistency is improved, so that the target three-dimensional model obtained based on multiple target single-perspective images is more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 A schematic diagram of a 3D model editing scene provided for this application;

[0019] Figure 2 A schematic diagram of a three-dimensional model editing method provided in Example 1;

[0020] Figure 3 A schematic diagram of a three-dimensional model editing method provided in Example 2;

[0021] Figure 4 A schematic diagram of a three-dimensional model editing method provided in Example 6;

[0022] Figure 5 A schematic diagram of a three-dimensional model editing method provided in Example 7;

[0023] Figure 6 A schematic diagram of a three-dimensional model editing method provided in Example 10;

[0024] Figure 7 A schematic diagram of a three-dimensional model editing method provided in Example 11;

[0025] Figure 8 A schematic diagram of a three-dimensional model editing method provided in Example 12;

[0026] Fig. 9 A schematic diagram of the overall flow of a three-dimensional model editing method provided in Example 13;

[0027] Fig.10 A schematic diagram of the structure of a three-dimensional model editing device provided in Example 14;

[0028] Fig.11 This is a schematic diagram of the structure of the editing device provided in Example 14. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0030] It should be noted that, in the description of this application, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. The terms "first", "second", etc. in this application are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0031] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0032] In conjunction with the specific application environment architecture or the specific hardware architecture on which the execution of the three-dimensional model editing method depends, the specific application environment architecture or the specific hardware architecture is described herein.

[0033] Nowadays, three-dimensional dynamic changes are widely used in augmented / virtual reality, film / game production, and artistic creation, so how to achieve high-quality three-dimensional dynamic changes is particularly important.

[0034] In the related art, the editor can perform the same type of replacement or minor feature changes on objects in the source 3D model, such as color changes. However, it cannot handle state changes.

[0035] In order to solve the defects of the related art, the inventor of this solution designed a new solution after creative research. The present solution provides a three-dimensional model editing method. In order to solve the problem that only partial feature changes of the target object can be achieved in the same state or the target object is replaced by an object of the same type in the related art, the present application realizes non-rigid editing based on text drive. Specifically, the present application obtains multiple source images under different perspectives from the source three-dimensional model, and then determines the initial text prompt information and the target text prompt information, wherein the target text prompt information includes the target state information of the target object, that is, the target state is formed after the target object performs the target action from the source state, so the present application can realize the change of the state of the target object, and then realizes the non-rigid editing of the source image, and increases the application scenario; in order to solve the problem of accuracy, the present application can obtain multiple source images under different perspectives, for each source image, a single-perspective image editing model that has been trained to convergence is used, and the target single-perspective image corresponding to the source image is determined based on the source text embedding data and the target text embedding data corresponding to the source image, and then the target three-dimensional model is reconstructed according to the multiple target single-perspective images. At this time, the target object in the target three-dimensional model is a three-dimensional model formed after the target action is performed. Since the source images under different perspectives are considered in the present application, the target three-dimensional model can be made more accurate. It should be noted that, in the present application, the single-view image editing model that has been trained to convergence edits each source image respectively to obtain the corresponding target single-view image after editing.

[0036] Figure 1 A schematic diagram of a 3D model editing scene provided for this application, such as Figure 1 As shown, the specific application scenario of the present application includes an editing device 101.

[0037] The editing device 101 may be a computer or a server, which is not limited here.

[0038] In this scenario, the editing device 101 obtains source images at different perspectives from the source three-dimensional model, and obtains target text prompt information, and respectively inputs the source image and the target text prompt information into a single-perspective image editing model that has been trained to convergence, and outputs the target single-perspective image at the corresponding perspective. This process is repeated to obtain target single-perspective images at multiple perspectives.

[0039] Furthermore, the editing device 101 reconstructs a target three-dimensional model based on multiple target single-view images.

[0040] In combination with the above scenarios, it can be seen that the related technology can realize the change of the remaining features of the target object in the same state, such as the change of color, but the change of the state of the target object cannot be realized. In this application, with the help of target text prompt information, the target text prompt information includes the target state information formed after the target object performs the target action, so that the source image can be non-rigidly edited according to the target text prompt information in the source state; in this application, a single-view image editing model that has been trained to convergence is used, and the target single-view image of each source image after editing at the corresponding perspective can be output respectively, thereby realizing non-rigid editing; since the single-view image editing model is trained to convergence, the target single-view image is more accurate, that is, it accurately represents the target state formed after the target object performs the target action from the source state; in addition, since different perspectives are taken into account in this application, the target three-dimensional model reconstructed based on multiple target single-view images is more accurate.

[0041] The embodiment of the present application provides a three-dimensional model editing method, and the method is described in detail in conjunction with the execution flow of the three-dimensional model editing method.

[0042] Embodiment 1

[0043] The execution entity of Examples 1 to 13 of the present application is a three-dimensional model editing device (referred to as editing device), which is located in the editing equipment.

[0044] Figure 2 A schematic diagram of a three-dimensional model editing method provided in Example 1. Figure 2 As shown, specifically including:

[0045] S201, obtaining multiple source images at different viewing angles from a source three-dimensional model, where the source images are images corresponding to the three-dimensional model at different viewing angles; each source image includes a target object; and the target object is in a source state.

[0046] The source three-dimensional model refers to a three-dimensional model of a target object to be edited.

[0047] The source image refers to the image corresponding to the source three-dimensional model at different viewing angles. It should be noted that the source image is two-dimensional.

[0048] The viewing angle may refer to the side, front or back, etc., which is not limited here. It should be noted that if the viewing angle is the side, then a side source image is an image corresponding to the source 3D model in the front.

[0049] The target object refers to an object included in the source 3D model or the source image. For example, if the source 3D model includes a cat, the cat is the target object.

[0050] The source state refers to the posture of the target object in the source 3D model or the source image. For example, if the source 3D model includes a standing cat, the source state is "standing". It should be noted that the source state of the target object in the source 3D model and the source image is consistent.

[0051] In one embodiment, the present application uses rendering technology to render source images from different perspectives from the source 3D model, for example, a source image from the front, a source image from the left side, a source image from the right side, and a source image from the back.

[0052] It should be noted that, in each source image, the target object is only at a different perspective, but the source state of the target object is the same.

[0053] S202, determining initial text prompt information corresponding to each source image and target text prompt information corresponding to each target single-view image to be generated, wherein the initial text prompt information includes source state information of the target object, and the target text prompt information includes target state information of the target object.

[0054] Among them, the initial text prompt information includes source state information of the target object, wherein the source state information includes information on the state of the target object, and the initial text prompt information also includes the size, color, etc. of the target object. For example, if the initial text prompt information includes a standing yellow cat, then the initial text prompt information is information such as "standing" and "yellow", and the source state information is "standing"; or, if the initial text prompt information includes a standing yellow kitten, then the initial text prompt information is information such as "standing", "yellow" and "small", and the source state information is "standing".

[0055] The target text prompt information includes target state information of the target object after executing the target action, wherein the target text prompt information refers to information such as the state, color and size of the target object, and the target state information includes posture information of the target object.

[0056] Among them, the target text prompt information includes a sitting yellow kitten, then the target text prompt information is information such as "sitting", "yellow" and "small", and the target state information is "sitting".

[0057] It should be noted that the target text prompt information and the initial text prompt information have common information, that is, the remaining information except the state of the target object can be considered common information. For example, the initial text prompt information is "a standing yellow kitten", and the target text prompt information is "a sitting yellow kitten", in which the target object only changes in state, and the remaining information does not change, such as "yellow" and "small", which is the common information.

[0058] In the present application, the editing device can change the target object from one state to another state without changing other features of the target object, that is, the color and size of the target object can remain unchanged, but the state of the target object changes. It can be understood that the target text prompt information is based on the initial text prompt information, and the generated target single-view image can keep other features of the target object unchanged and only change the state of the target object.

[0059] In one approach, the size, color or type of the target object may also be changed. It should be noted that color, size and type of the target object are also features of the image.

[0060] The target single-view image is an image obtained after the target object performs the target action and forms a target state from a source state.

[0061] S203, determining the source text embedding data corresponding to each source image according to each initial text prompt information; and determining the target text embedding data corresponding to each target single-view image to be generated according to each target text prompt information, each target single-view image is an image that forms a target state from a source state after a target object performs a target action under the perspective of the corresponding source image.

[0062] The perspectives of the target single-view image and the source image correspond to each other. For example, a front source image generates a front target single-view image.

[0063] The source text embedding data and the target text embedding data are obtained after encoding.

[0064] It should be noted that the source text embedding data includes the source state embedding data of the target object, that is, it represents the source state of the source image; the target text embedding data includes the target state embedding data of the target object, that is, it represents the target state of the target single-view image.

[0065] S204, for each source image, using a single-view image editing model that has been trained to convergence and determining a target single-view image corresponding to the source image based on source text embedding data and target text embedding data corresponding to the source image, so as to achieve non-rigid editing.

[0066] Among them, the single-view image editing model that has been trained to convergence refers to a model that performs single-view editing on a source image based on target text prompt information, so that the target object in the source image changes from a source state to a target state under single-view control.

[0067] It should be noted that the single-view image editing model that has been trained to convergence processes each source image separately, thereby outputting the target single-view image corresponding to each source object in turn.

[0068] It should be noted that in one editing task, the single-view image editing model that has been trained to convergence receives a source image, edits the source image, and obtains a target single-view image at the corresponding perspective to complete this editing task. For the next source image at a different perspective, the target single-view image corresponding to the same method is obtained.

[0069] In this embodiment, the target single-view image is outputted by embedding the source text data, which can ensure that the non-rigid editing is completed under the original features of the source image, that is, the target single-view image is ensured to achieve non-rigid editing from the source state to the target state based on the original features of the source image. The original features may include the color, size and other information features of the target object in the source image.

[0070] S205, reconstructing a target three-dimensional model based on each target single-view image.

[0071] The target three-dimensional model includes a target object. At this time, the target object in the target three-dimensional model is in a target state, that is, a target state formed after the target action is executed.

[0072] The target three-dimensional model refers to a three-dimensional model formed after the target object in the source three-dimensional model performs a target action.

[0073] Method 1: A 3D model is constructed based on the corresponding single-view images of the target under different viewing angles, which is the target 3D model.

[0074] Method 2: Specifically, the edited initial 3D model is initialized with the source 3D model, and the edited initial 3D model is rendered at a preset viewing angle to obtain a rendered image. , the target single-view image at a preset viewing angle is obtained from the target single-view images at different viewing angles obtained in step S204, and is recorded as ,Will as well as Input into the reconstruction loss function to calculate the iterative reconstruction loss value, optimize the edited initial 3D model based on the iterative reconstruction loss value, obtain the edited optimized 3D model, and complete the first iteration. Among them, the reconstruction loss function is ,in, refers to the image after non-rigid editing, is a rendered image, where Represents the iterative reconstruction loss value, where the source 3D model is consistent with the edited initial 3D model.

[0075] Further, for the edited optimized 3D model, the iteration is continued, specifically: rendering from any perspective of the edited optimized 3D model to obtain the rendered image in the second step of the iteration , the rendered image Input it into the single-view image editing model that has been trained to convergence, and output the edited image to obtain , continue to as well as The iterative reconstruction loss value is input into the reconstruction loss function, and the optimized three-dimensional model is continuously optimized based on the iterative reconstruction loss value, and the above steps are continuously iterated until the iterative reconstruction loss value or the number of iterations corresponding to a certain three-dimensional model at any viewing angle meets the preset iteration condition, and the three-dimensional model at this time is determined to be the target three-dimensional model. The preset iteration condition includes the preset reconstruction loss value and / or the preset number of iterations.

[0076] Exemplarily, if the calculated iterative reconstruction loss value corresponding to any viewing angle is less than a preset reconstruction loss value, it is determined that the preset iteration condition is met; or, if the number of iterations for the three-dimensional model has reached a preset number of iterations, it is determined that the preset iteration condition is met.

[0077] In the second method, when iteratively optimizing the three-dimensional model, iterative optimization can be performed from different perspectives, thereby combining multiple perspectives to make the target three-dimensional model more accurate and reasonable.

[0078] It should be noted that in the present application, the target state can be a local state change of the target object from the source state, or a global state change of the target object from the source state. It can be seen that in the present application, global or local non-rigid editing of the target object can be achieved; the editing device in the present application achieves global or local non-rigid editing of the source image, that is, it can enable the target object in the source image to achieve global or local state change non-rigid editing.

[0079] It should be noted that the editing device of the present application can also achieve changes in the color and size of the target object in the source image. Exemplarily, in one method, in addition to the inconsistency between the state of the target object and the source state, the target text prompt information can also include features to be changed. For example, the initial text prompt information includes "a sitting yellow kitten", and the target text prompt information includes "a standing red kitten". From the source image to the target single-view image, not only the state of the target object changes, but also the color changes, wherein the color is the feature to be changed. Therefore, in this method, a single-view image editing model that has been trained to convergence is used to determine the target single-view image corresponding to the source image, and the state and color of the target object in the target single-view image are changed based on the source image.

[0080] The present embodiment provides a three-dimensional model editing method. In the present embodiment, first, source images corresponding to different perspectives are obtained from a source three-dimensional model, wherein the source images include a target object, and the target object is in a source state. Then, initial text prompt information and target text prompt information corresponding to each source image are determined, wherein the initial text prompt information includes source state information of the target object, and the target text prompt information includes target state information of the target object. Then, source text embedding data of the corresponding source image is determined according to each initial text prompt information. Further, target text embedding data is determined according to the target text prompt information to generate a target single-perspective image, wherein the target single-perspective image is an image that forms a target state from a source state after the target object performs a target action under the perspective of the corresponding source image. Further, according to each source image, a single-perspective image editing model that has been trained to convergence is used to determine the corresponding target single-perspective image based on the source text embedding data and the target text embedding data corresponding to the source image, so that the edited target single-perspective image corresponding to each source image can be obtained, thereby editing and reconstructing the target three-dimensional model based on each target single-perspective image. It can be seen that in the present application, with the help of the target text prompt information, the source image can perform the target action in the corresponding source state to form a target single-view image of the target state. The target state in the present application can be a local state change of the target object from the source state, or a global state change of the target object from the source state. It can be seen that non-rigid editing of the target object can be achieved in the present application; in addition, the present application can achieve non-rigid editing under the action of the target text prompt information, and then achieve text-driven non-rigid editing of the image; in the present application, for the source three-dimensional model, source images are obtained from multiple perspectives respectively, and edited target single-view images are obtained from multiple source images respectively. Since the present application considers multiple perspectives, the target three-dimensional model obtained based on multiple target single-view images is more accurate.

[0081] Embodiment 2

[0082] This embodiment is a further refinement of any of the above embodiments. This embodiment is an optional way to determine the initial text prompt information corresponding to each source image.

[0083] Figure 3 A schematic diagram of a three-dimensional model editing method provided in Example 2. Figure 3 As shown, specifically including:

[0084] S301, inputting each source image into a preset text prediction algorithm respectively, and using the preset text prediction algorithm to determine image description information of the source image respectively.

[0085] Among them, the preset text prediction algorithm can be a multimodal algorithm.

[0086] Specifically, each source image is input into a preset text prediction algorithm, thereby outputting image description information of the source image.

[0087] The image description information includes a description of the source image features, such as the color, size, and posture description information of the target object.

[0088] S302: Determine the image description information of each source image as each initial text prompt information.

[0089] This embodiment provides a three-dimensional model editing method. In this embodiment, a preset text prediction algorithm can be used to output image description information corresponding to each source image, thereby accurately obtaining the image description information.

[0090] Embodiment 3

[0091] This embodiment is a further refinement of any of the above embodiments. This embodiment is an optional method for determining target text prompt information corresponding to each target single-view image to be generated, including:

[0092] According to each initial text prompt information and the target state, the target text prompt information is generated.

[0093] The initial text prompt information includes the color, size and viewing angle information of the target object and the corresponding source image, so that the editing device can generate the target text prompt information based on the initial text prompt information and in combination with the target state.

[0094] Therefore, the target text prompt information may include the target object color, size, corresponding viewing angle, and target state information.

[0095] It should be noted that the initial text prompt information and the target text prompt information may also include color, size, corresponding viewing angle information, target object type, target state and other feature information, which are not limited here.

[0096] It should be noted that the target text prompt information may include the target object color, size, target state information and viewing angle information. Among them, the target object color and size can be obtained from the initial text prompt information. Among them, the viewing angle information is configuration information, and the viewing angle information can be "perform the target action at the viewing angle corresponding to the source image", that is, control the execution of the target action at the viewing angle corresponding to the source image to form the target state from the source state. If the color and size of the target object in the target text prompt information remain unchanged, the target text prompt information represents the control to ensure that the color and size of the target object remain unchanged at the viewing angle corresponding to the source image, and execute the target action to form the target state.

[0097] This embodiment provides a three-dimensional model editing method. In this embodiment, target text prompt information is generated based on initial text prompt information and target state, so that the target text prompt information includes part of the information in the initial text prompt information, thereby enabling the source image to perform the target action under the original features to form the target state.

[0098] Embodiment 4

[0099] This embodiment is a further refinement of any of the above embodiments. This embodiment is an optional method for determining source text embedding data corresponding to each source image according to each initial text prompt information, including:

[0100] Step 1: Input each initial text prompt information into a preset text editor, and use the preset text editor to encode each initial text prompt information respectively to obtain each initial text embedding data.

[0101] Among them, the preset text editor is a pre-set text editor.

[0102] Among them, the initial text embedding data is available Represented by , where m is the number of tokens in the text encoding and d is the dimension of each token.

[0103] It should be noted that text embedding data is the conversion of text data into high-dimensional, dense vector data, which can measure the semantic and syntactic similarities between different texts.

[0104] Step 2: Optimize the initial text embedding data to obtain the embedding data of each source text that meets the optimization conditions.

[0105] The optimization condition refers to a pre-set condition for optimizing the initial text embedding data.

[0106] This embodiment provides a three-dimensional model editing method. In this embodiment, a preset text editor is used to accurately obtain initial text embedding data.

[0107] Embodiment 5

[0108] This embodiment is a further refinement of any of the above embodiments. This embodiment is an optional method for determining target text embedding data corresponding to each target single-view image to be generated according to each target text prompt information, including:

[0109] Input each target text prompt information into a preset text editor, and use the preset text editor to encode each target text prompt information respectively, so as to obtain each target text embedded data.

[0110] In this embodiment, the target text embedding data can be used express.

[0111] This embodiment provides a three-dimensional model editing method. In this embodiment, a preset text editor is used to accurately obtain target text embedding data.

[0112] Embodiment 6

[0113] This embodiment is a further refinement of any of the above embodiments. This embodiment is an optional method for realizing non-rigid editing by using a single-view image editing model that has been trained to convergence for each source image and determining the target single-view image corresponding to the source image based on the source text embedding data and the target text embedding data corresponding to the source image.

[0114] Figure 4 A schematic diagram of a three-dimensional model editing method provided in Example 6. Figure 4 As shown, including:

[0115] S401, determining optimized target text embedding data based on source text embedding data corresponding to a source image and target text embedding data.

[0116] In one approach, for each source image, the corresponding source text embedding data and target text embedding data are combined to obtain optimized target text embedding data.

[0117] It should be noted that the target text embedding data represents the features corresponding to the source image and also combines the information included in the target text prompt information.

[0118] S402, respectively inputting the optimized target text embedding data corresponding to each source image into the single-view image editing model that has been trained to convergence, and using the single-view image editing model that has been trained to convergence to respectively determine the corresponding target single-view image under each perspective, so as to realize non-rigid editing.

[0119] Specifically, for each source image, assuming that there are four source images under different perspectives, namely, a front source image, a left side source image, a right side source image, and a back source image, then based on the above steps, the optimized target text embedding data corresponding to the four source images can be obtained respectively, namely, the first optimized target text embedding data, the second optimized target text embedding data, the third optimized target text embedding data, and the fourth optimized target text embedding data.

[0120] Further, the first optimized target text embedding data is input into a single-view image editing model that has been trained to convergence to output a target single-view image corresponding to the front source image. Exemplarily, it is assumed that the target text prompt information includes the same target object color, size, perspective information and target state in the source image, and the target state is inconsistent with the source state in the initial text prompt information. Furthermore, the edited target single-view image is an image in which the target object color and size are consistent with the source image in the front, and the target object state changes from the source state to the target state.

[0121] The present embodiment provides a three-dimensional model editing method. In the present embodiment, target single-view images output by a single-view image editing model that has been trained to convergence can be obtained respectively for each source image. Since the single-view image editing model has been trained to convergence, the target single-view image is more accurate. In addition, the present embodiment adopts optimized target text embedding data, which includes semantic commonalities and semantic differences. Thus, the present embodiment takes semantic compression into consideration, so that in the non-rigid editing process, not only the guidance of the semantic difference part on the non-rigid editing is considered, but also the semantic commonality is used to maintain the original features of the source image, thereby achieving controllable non-rigid editing based on the source image, and obtaining a more accurate target single-view image.

[0122] Embodiment 7

[0123] This embodiment is a further refinement of any of the above embodiments. This embodiment is an optional way to determine optimized target text embedding data based on source text embedding data and target text embedding data corresponding to a source image.

[0124] Figure 5 A schematic diagram of a three-dimensional model editing method provided in Example 7. Figure 5 As shown, including:

[0125] S501, decomposing the target text embedding data based on the source text embedding data corresponding to the source image to obtain a projection component and a vertical component, wherein the projection component represents the component of semantic commonality between the source text embedding data and the target text embedding data, and the vertical component represents the component of semantic difference between the source text embedding data and the target text embedding data.

[0126] S502, determining optimized target text embedding data based on the target text embedding data, the projection component and the vertical component.

[0127] In this embodiment, the projection component and the vertical component are calculated. Since the projection component can represent semantic commonalities, and thus can be combined with the semantic commonalities of the source text embedding data and the target text embedding data, and the vertical component can represent semantic differences, and thus can be combined with the semantic differences of the source text embedding data and the target text embedding data, the determined optimized target text embedding data takes into account the semantic commonalities and semantic differences.

[0128] The present embodiment provides a three-dimensional model editing method. In the present embodiment, the source text embedding data and the target text embedding data are decomposed to calculate the projection component and the vertical component, and then the optimized target text embedding data is calculated. Since the projection component considers semantic commonalities and the vertical component considers semantic differences, the semantic commonalities and differences are considered in the optimized target text embedding data, thereby making the optimized target text embedding data more accurate.

[0129] Embodiment 8

[0130] This embodiment is a further refinement of any of the above embodiments. This embodiment is an optional method of decomposing target text embedding data based on source text embedding data corresponding to a source image to obtain a projection component and a vertical component, including:

[0131] The source text embedding data and the target text embedding data are input into a preset text embedding fusion algorithm to output a projection component and a vertical component.

[0132] Exemplarily, the preset text embedding fusion algorithm is shown as (1):

[0133] (1)

[0134] in, represents the vector inner product, represents the projection component, represents the vertical component, Indicates that the source text is embedded with data, Represents the target text embedding data.

[0135] Specifically, the source text embedding data and the target text embedding data are input into a preset text embedding fusion algorithm, and the projection component and the vertical component are obtained through vector inner product.

[0136] This embodiment provides a three-dimensional model editing method. In this embodiment, a preset text embedding fusion algorithm is used to calculate the projection component and the vertical component respectively through the vector inner product.

[0137] Embodiment 9

[0138] This embodiment is a further refinement of any of the above embodiments. This embodiment is an optional method for determining optimized target text embedding data based on target text embedding data, projection components and vertical components, including:

[0139] The target text embedding data, the projection component and the vertical component are input into a preset text embedding optimization algorithm to output the optimized target text embedding data.

[0140] Among them, the preset text embedding optimization algorithm is shown as (2):

[0141] (2)

[0142] in, Represents the optimized target text embedding data; is the semantic difference parameter, which is a pre-set weight coefficient used to control the importance of semantic difference; W is the semantic commonality parameter, which is used to control the importance of semantic commonality. The larger W is, the smaller the influence of semantic commonality is.

[0143] In this embodiment, interpolation may be used to obtain optimized target text embedding data.

[0144] This embodiment provides a three-dimensional model editing method. In this embodiment, a preset text embedding optimization algorithm is used to calculate the optimized target text embedding data based on the target text embedding data, the projection component and the vertical component, so that the optimized target text embedding data is more accurate.

[0145] Embodiment 10

[0146] This embodiment is a further refinement of any of the above embodiments. In this embodiment, the single-view image editing model that has been trained to convergence is a model in which a low-rank adaptive layer is added to the key vector and value vector of the cross-attention module of the single-view image generation model that has been trained to convergence.

[0147] The single-view image generation model that has been trained to convergence is used to prepare for recognition of the source image, and can read out the original features in the source image, such as the target object type, target object color, target object size, and source state of the target object, and generate an edited image to be optimized based on the original features of the source image, and the edited image to be optimized is similar to the features in the source image. Since it is a single-view image generation model that has been trained to convergence, it is obtained after training, so that the edited image is similar to the features in the source image.

[0148] This embodiment is an optional method for achieving non-rigid editing before using a single-view image editing model that has been trained to convergence for each source image and determining the target single-view image corresponding to the source image based on the source text embedding data and the target text embedding data corresponding to the source image.

[0149] Figure 6 A schematic diagram of a three-dimensional model editing method provided in Example 10. Figure 6 As shown, including:

[0150] S601, obtaining a single-view image generation model that has been trained to convergence.

[0151] The single-view image generation model that has been trained to convergence may be one that is obtained by continuing to control input and its own training parameters on the basis of a basic diffusion model that has been trained to convergence.

[0152] S602, keeping the single-view image generation model that has been trained to convergence unchanged, and training the training parameters of the low-rank adaptive layer until a first convergence condition is reached.

[0153] It should be noted that in this embodiment, a low-rank adaptive layer is added to the key vector and the value vector based on the single-view generation model that has been trained to convergence, so that only the training parameters in the low-rank adaptive layer need to be adjusted, and the parameters in the single-view image generation model that has been trained to convergence are not trained, so as to keep the single-view image generation model that has been trained to convergence unchanged, and the training image in the training sample is input into the low-rank adaptive layer, and the training parameters in the low-rank adaptive layer are adjusted until the adjusted training parameters make the edited image output from the single-view image editing model consistent with the actual image in the training sample, and then it is determined that the first convergence condition is met. Among them, the training sample includes the training image and its corresponding actual image, wherein the actual image refers to an image that forms a certain state after the training object performs the training action based on the training features of the training image. The training object in the training image is in the previous state, and the training object in the actual image is in the later state.

[0154] The first convergence condition refers to a convergence condition under which the training parameters of the low-rank adaptive layer are adjusted so that the edited image reaches a certain set consistency with the actual image in the training sample.

[0155] S603: Determine the single-view image editing model that meets the first convergence condition as the single-view image editing model that has been trained to convergence.

[0156] It should be noted that when the first convergence condition is reached, the training parameters of the low-rank adaptive layer are adjusted and adjusted to the target parameters, and the single-view image editing model corresponding to the target parameters at this time is determined as the single-view image editing model that has been trained to convergence.

[0157] This embodiment provides a three-dimensional model editing method. In this embodiment, during the training process, the single-view image generation model that has been trained to convergence is kept unchanged, and only the training parameters of the low-rank adaptive layer are adjusted, thereby reducing the amount of optimization parameters and speeding up the network training speed.

[0158] Embodiment 11

[0159] This embodiment is a further refinement of any of the above embodiments. This embodiment is an optional method of the training process of the single-view image generation model that has been trained to convergence as a diffusion model.

[0160] Figure 7 A schematic diagram of a three-dimensional model editing method provided in Example 11. Figure 7 As shown, including:

[0161] S701, obtaining an initial single-view image generation model.

[0162] The initial single-view image generation model may be a basic diffusion model that has been trained to convergence.

[0163] S702, keeping the source text embedding data corresponding to the source image unchanged, inputting the source text embedding data into the initial single-view image generation model, and training the training parameters of the initial single-view image generation model until a second convergence condition is reached.

[0164] For example, the training process of the single-view image generation model can be shown as (3):

[0165] (3)

[0166] in, is the noisy input image, that is, the image formed by adding noise to the source image; is the true value of the noise, represents the single-view image generation model, represents the training parameter weights of the single-view image generation model, is the source text embedding data, t is the time step of the single-view image generation model, Refers to the diffusion loss value when training a single-view image generation model.

[0167] In this step, the input source text embedding data is kept unchanged and input into (3), and the training parameters of the initial single-view image generation model are trained until the adjusted training parameters make the image to be optimized output by the single-view image generation model and the source image meet the second convergence condition.

[0168] Among them, the second convergence condition refers to the training parameters adjusted so that the consistency between the image to be optimized output by the single-view image generation model and the source image is greater than or equal to the first preset consistency. For example, assuming that the first preset consistency is 95%, the consistency between the image to be optimized and the source image is greater than or equal to 95%, that is, the second convergence condition is met.

[0169] It should be noted that, in order to ensure accuracy, the first preset consistency can be set to a larger value, for example, to 98% or 100%, and there is no limitation here.

[0170] S703: Determine the single-view image generation model that meets the second convergence condition as the single-view image generation model that has been trained to convergence.

[0171] Exemplarily, if the training parameters of the single-view image generation model meet the second convergence condition after adjustment, the single-view image generation model corresponding to the adjusted training parameters is determined as the single-view image generation model that has been trained to convergence.

[0172] The present embodiment provides a three-dimensional model editing method. In the present embodiment, an initial single-view image generation model is first obtained, and then the source text embedding data is kept unchanged, the source text embedding data is input into the initial single-view image generation model, and the training parameters therein are trained until the second convergence condition is reached, thereby determining the single-view image generation model that has been trained to convergence. The present application uses the source image to train the single-view image generation model, so as to improve generalization; in addition, the single-view image generation model in the present embodiment is trained with the source image to maintain the original features of the source image.

[0173] Embodiment 12

[0174] This embodiment is a further refinement of any of the above embodiments. This embodiment is an optional way to optimize the initial text embedding data to obtain each source text embedding data that meets the optimization conditions.

[0175] Figure 8 This is a schematic diagram of a three-dimensional model editing method provided in Example 12. Figure 8 As shown, including:

[0176] S801, inputting the initial text embedding data into the initial single-view image generation model to output the image to be optimized.

[0177] Exemplarily, the process of optimizing the initial text embedding data is shown in (4):

[0178] (4)

[0179] in, is the noisy input image, that is, the image formed by adding noise to the source image; is the true value of the noise, represents the single-view image generation model, represents the training parameter weights of the single-view image generation model, represents text embedding data, t is the time step of the single-view image generation model, Represents the diffusion loss value under the initial single-view image generation model.

[0180] It should be noted that, in step S801, Generate a model for the initial single-view image, It includes initial text embedding data and new text embedding data after adjusting the initial text embedding data.

[0181] In this step, the initial text embedding data is input into (4), and the initial single-view image generation model is used to output the image to be optimized.

[0182] S802: In response to the image to be optimized satisfying the optimization condition, determining that the initial text embedding data is the source text embedding data.

[0183] The optimization condition may be a condition that the consistency between the image to be optimized and the source image is greater than or equal to a second preset consistency. The second preset consistency is less than the first preset consistency. For example, assuming that the first preset consistency is 98%, the second preset consistency may be set to 95%.

[0184] Exemplarily, assuming that the consistency between the image to be optimized and the source image is 96%, it is determined that the image to be optimized meets the optimization condition, and the initial text embedding data is determined to be the source text embedding data.

[0185] It should be noted that the illustrative examples in this application are only used to illustrate the scheme and do not represent the actual situation.

[0186] S803, in response to the image to be optimized not satisfying the optimization condition, continue to adjust the initial text embedding data, and keep the initial single-view image generation model unchanged, until the output image to be optimized satisfies the optimization condition, and determine the text embedding data corresponding to the optimization condition when the optimization condition is satisfied as the source text embedding data.

[0187] Exemplarily, if the consistency between the image to be optimized and the source image is 80%, which is determined to be less than the second preset consistency (95%), it is determined that the optimization condition is not met.

[0188] Further, continue to adjust the initial text embedding data , thereby obtaining new text embedding data , and keep the initial single-view image generation model, that is, keep the training parameters unchanged, and then input the new text embedding data into (4), so as to output a new image to be optimized, and continue to determine whether the image to be optimized meets the optimization conditions. If not, continue to adjust the initial text embedding data until the adjusted text embedding data makes the image to be optimized output by the initial single-view image generation model meet the optimization conditions, and determine the text embedding data corresponding to the optimization conditions at this time as the source text embedding data.

[0189] The present embodiment provides a three-dimensional model editing method. In the present embodiment, the initial single-view image generation model remains unchanged, and the initial text embedding data is adjusted so that the output image to be optimized meets the optimization conditions, and then the corresponding text embedding data when the optimization conditions are met is determined as the source text embedding data. Therefore, a text embedding optimization strategy is proposed in the present embodiment. By optimizing the initial text embedding data, accurate source text embedding data is obtained, and the source text embedding data can more accurately describe the source image.

[0190] Embodiment 13

[0191] Fig. 9 This is a schematic diagram of the overall flow of a three-dimensional model editing method provided in Example 13. Fig. 9 As shown, it includes: a source three-dimensional model 901, a source image 902, a single-view image editing model 903 that has been trained to convergence, a target single-view image 904, and a target three-dimensional model 905.

[0192] exist Fig. 9 In the example, the target object in the source 3D model 901 is a cat, and the target objects have their own corresponding features. A front source image 902 is rendered from the source 3D model 901, and the source image 902 is input into a single-view image editing model 903 that has been trained to convergence, and a target single-view image 904 corresponding to the front source image 902 under the source image perspective is output based on the target text prompt information.

[0193] It should be noted that the source images 902 at other perspectives can continue to be obtained from the source 3D model 901, and the target single-view image 904 at the corresponding perspective can be output using the single-view image editing model 903 that has been trained to convergence.

[0194] Furthermore, a target three-dimensional model is reconstructed based on multiple target single-view images 904 .

[0195] It should be noted that the single-view image generation model that has been trained to convergence in the present application is trained through source images. There can be multiple source images. The single-view image generation model can be trained each time a source image is obtained to obtain a single-view image generation model that has been trained to convergence; or, a single-view image generation model that has been trained to convergence can be obtained by using one source image, and the remaining source images can use the above-mentioned one source image to obtain the single-view image generation model that has been trained to convergence to output the target single-view image.

[0196] It should be noted that text embedding data fusion is implemented in the present application, that is, the source text embedding data and the target text embedding data are fused to obtain optimized target embedding data. In addition, the source image is used in the present application to implement the training of the single-view image generation model, and a single-view image generation model that has been trained to convergence is obtained. In addition, a low-rank adaptive layer is added to the single-view image generation model that has been trained to convergence, and only the training parameters of the low-rank adaptive layer are trained to obtain a single-view image editing model that has been trained to convergence. The single-view image editing model that has been trained to convergence can edit the source image into a target single-view image, thereby completing non-rigid editing. In addition, the target state in the target single-view image can be a local state change or a global state change, thereby achieving global or local non-rigid editing.

[0197] It should be noted that the present application proposes to extract features from source images based on text embedding data optimization to obtain source text embedding data, wherein the source text embedding data includes the original characteristics of the source image.

[0198] It should be noted that in the present application, optimized target text embedding data is obtained based on the source text embedding data and the target text embedding data, so that the target single-view image can complete the change from the source state to the target state without changing the original features of the source image, thereby completing non-rigid editing.

[0199] It should be noted that since the single-view image editing model refers to a single-view image generation model that has been trained to convergence and a low-rank adaptive layer is added, when training the single-view image editing model in this application, only the low-rank adaptive layer needs to be trained, which can speed up the training speed.

[0200] It should be noted that the semantically compressed text embedding fusion strategy proposed in this application is based on the decomposition of the source text embedding data and the target text embedding data to obtain the components of semantic commonality and the components of semantic difference, so that in the process of non-rigid editing, the guidance of semantic difference on non-rigid editing is taken into account, and at the same time, the original features of the source image are maintained by using semantic commonality, thereby realizing non-rigid editing.

[0201] In summary, the present application proposes text-driven non-rigid editing of the source three-dimensional model, thereby achieving global or local three-dimensional non-rigid editing, thereby providing accuracy, consistency and controllability of the editing results.

[0202] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method.

[0203] Embodiment 14

[0204] The following is an embodiment of the device of the present application. Fig.10 This is a schematic diagram of the structure of a three-dimensional model editing device provided in Example 14. Fig.10 As shown, the 3D model editing device 1000 includes the following modules:

[0205] The acquisition module 1001 is used to acquire multiple source images at different viewing angles from the source three-dimensional model, where the source images are images corresponding to the three-dimensional model at different viewing angles; each source image includes a target object; and the target object is in a source state;

[0206] A determination module 1002 is used to determine initial text prompt information corresponding to each source image and target text prompt information corresponding to each target single-view image to be generated, wherein the initial text prompt information includes source state information of the target object, and the target text prompt information includes target state information of the target object;

[0207] The determination module 1002 is further used to determine the source text embedding data corresponding to each source image according to each initial text prompt information; the determination module 1002 is further used to determine the target text embedding data corresponding to each target single-view image to be generated according to each target text prompt information, each target single-view image is an image formed from a source state to a target state after the target object performs a target action under the perspective of the corresponding source image;

[0208] The determination module 1002 is further configured to determine, for each source image, a target single-view image corresponding to the source image using a single-view image editing model that has been trained to convergence and based on source text embedding data and target text embedding data corresponding to the source image, so as to achieve non-rigid editing;

[0209] The reconstruction module 1003 is used to reconstruct the target 3D model based on each target single-view image.

[0210] Optionally, when determining the initial text prompt information corresponding to each source image, the determination module 1002 is specifically configured to:

[0211] Inputting each source image into a preset text prediction algorithm, and using the preset text prediction algorithm to determine image description information of each source image;

[0212] The image description information of each source image is determined as each initial text prompt information.

[0213] The determination module 1002, when determining the target text prompt information corresponding to each target single-view image to be generated, is specifically used to:

[0214] According to each initial text prompt information and the target state, the target text prompt information is generated.

[0215] The determination module 1002, when determining the source text embedding data corresponding to each source image according to each initial text prompt information, is specifically used to:

[0216] Input each initial text prompt information into a preset text editor, and use the preset text editor to encode each initial text prompt information respectively to obtain each initial text embedding data;

[0217] The initial text embedding data is optimized to obtain each source text embedding data when the optimization condition is satisfied.

[0218] The determination module 1002, when determining the target text embedding data corresponding to each target single-view image to be generated according to each target text prompt information, is specifically used to:

[0219] Input each target text prompt information into a preset text editor, and use the preset text editor to encode each target text prompt information respectively, so as to obtain each target text embedded data.

[0220] The determination module 1002, when determining the target single-view image corresponding to the source image based on the source text embedding data and the target text embedding data corresponding to the source image for each source image to implement non-rigid editing, is specifically used to:

[0221] Determine optimized target text embedding data based on source text embedding data corresponding to the source image and target text embedding data;

[0222] The optimized target text embedding data corresponding to each source image are respectively input into the single-view image editing model that has been trained to convergence, and the single-view image editing model that has been trained to convergence is used to determine the corresponding target single-view image under each perspective to achieve non-rigid editing.

[0223] The determination module 1002, when determining the optimized target text embedding data based on the source text embedding data corresponding to the source image and the target text embedding data, is specifically used to:

[0224] Decomposing the target text embedding data based on the source text embedding data corresponding to the source image to obtain a projection component and a vertical component, wherein the projection component represents the component of semantic commonality between the source text embedding data and the target text embedding data, and the vertical component represents the component of semantic difference between the source text embedding data and the target text embedding data;

[0225] The optimized target text embedding data is determined based on the target text embedding data, the projection component and the vertical component.

[0226] The determination module 1002, when decomposing the target text embedding data based on the source text embedding data corresponding to the source image to obtain the projection component and the vertical component, is specifically used to:

[0227] The source text embedding data and the target text embedding data are input into a preset text embedding fusion algorithm to output a projection component and a vertical component.

[0228] The determination module 1002, when determining the optimized target text embedding data based on the target text embedding data, the projection component and the vertical component, is specifically used to:

[0229] The target text embedding data, the projection component and the vertical component are input into a preset text embedding optimization algorithm to output the optimized target text embedding data.

[0230] The single-view image editing model trained to convergence is a model in which a low-rank adaptation layer is added to the key vector and value vector of the cross-attention module of the single-view image generation model trained to convergence;

[0231] The determination module 1002 is further used to determine the target single-view image corresponding to the source image based on the source text embedding data and the target text embedding data corresponding to the source image for each source image, so as to implement non-rigid editing, before adopting the single-view image editing model that has been trained to convergence and determining the target single-view image corresponding to the source image based on the source text embedding data and the target text embedding data corresponding to the source image, for example:

[0232] Obtain a single-view image generation model that has been trained to convergence;

[0233] Keeping the single-view image generation model that has been trained to convergence unchanged, the training parameters of the low-rank adaptive layer are trained until the first convergence condition is reached;

[0234] The single-view image editing model that reaches the first convergence condition is determined as the single-view image editing model that has been trained to convergence.

[0235] The single-view image generation model that has been trained to convergence is a diffusion model. This embodiment provides a three-dimensional model editing device, which also includes: a training module;

[0236] The training module is used during the training process of the single-view image generation model that has been trained to convergence to:

[0237] Obtain an initial single-view image generation model;

[0238] Keeping the source text embedding data corresponding to the source image unchanged, inputting the source text embedding data into the initial single-view image generation model, and training the training parameters of the initial single-view image generation model until a second convergence condition is reached;

[0239] The single-view image generation model that reaches the second convergence condition is determined as the single-view image generation model that has been trained to convergence.

[0240] The determination module 1002, when optimizing the initial text embedding data to obtain each source text embedding data satisfying the optimization condition, is specifically used for:

[0241] Input the initial text embedding data into the initial single-view image generation model to output the image to be optimized;

[0242] In response to the image to be optimized satisfying the optimization condition, determining the initial text embedding data as the source text embedding data;

[0243] In response to the image to be optimized not meeting the optimization conditions, the initial text embedding data continues to be adjusted, and the initial single-view image generation model is kept unchanged until the output image to be optimized meets the optimization conditions, and the text embedding data corresponding to the optimization conditions is determined as the source text embedding data.

[0244] The description of the features in the embodiment corresponding to the three-dimensional model editing device can refer to the relevant description of the embodiment corresponding to the three-dimensional model editing method, which will not be repeated here.

[0245] The embodiment of the present application also provides an editing device, Fig.11 This is a schematic diagram of the structure of the editing device provided in Example 14. Fig.11 As shown, the editing device 1100 includes a processor 1101 and a memory 1102, wherein the processor 1101, the memory 1102 and the communication component 1103 are connected via a bus 1104. The memory 1102 stores a computer program, and the processor 1101 is configured to run the computer program to execute the steps in any of the above three-dimensional model editing method embodiments.

[0246] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any one of the above-mentioned three-dimensional model editing method embodiments when running.

[0247] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0248] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any one of the above-mentioned three-dimensional model editing method embodiments are implemented.

[0249] An embodiment of the present application also provides another computer program product, including a non-volatile computer-readable storage medium, the non-volatile computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the steps in any of the above-mentioned three-dimensional model editing method embodiments are implemented.

[0250] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0251] The above is a detailed introduction to a three-dimensional model editing method provided by the present application. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and its core idea of ​​the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A three-dimensional model editing method, characterized in that: include: Acquire multiple source images at different viewing angles from a source three-dimensional model, wherein the source images are images corresponding to the three-dimensional model at different viewing angles; each source image includes a target object; and each target object is in a source state; Determine initial text prompt information corresponding to each of the source images and target text prompt information corresponding to each of the target single-view images to be generated, wherein the initial text prompt information includes source state information of the target object, and the target text prompt information includes target state information of the target object; Determine the source text embedding data corresponding to each source image according to each initial text prompt information; and determine the target text embedding data corresponding to each target single-view image to be generated according to each target text prompt information, wherein each target single-view image is an image formed from the source state to the target state after the target object performs the target action at the perspective of the corresponding source image; For each source image, a single-view image editing model that has been trained to convergence is used to determine a target single-view image corresponding to the source image based on source text embedding data and target text embedding data corresponding to the source image, so as to achieve non-rigid editing; The target 3D model is reconstructed based on the single-view images of each target.

2. The method according to claim 1, characterized in that The determining of the initial text prompt information corresponding to each of the source images includes: Inputting each of the source images into a preset text prediction algorithm, and using the preset text prediction algorithm to determine image description information of the source images; The image description information of each source image is determined as each initial text prompt information.

3. The method according to claim 2, characterized in that Determine the target text prompt information corresponding to each target single-view image to be generated, including: Target text prompt information is generated according to each of the initial text prompt information and the target state.

4. The method according to claim 1, characterized in that: The step of determining the source text embedding data corresponding to each source image according to each initial text prompt information includes: Inputting each of the initial text prompt information into a preset text editor, and using the preset text editor to encode each of the initial text prompt information respectively to obtain each initial text embedding data; The initial text embedding data is optimized to obtain each source text embedding data that meets the optimization condition.

5. The method according to claim 1, characterized in that The step of determining target text embedding data corresponding to each target single-view image to be generated according to each target text prompt information includes: The target text prompt information is input into a preset text editor, and the preset text editor is used to encode the target text prompt information respectively, so as to obtain the embedded data of each target text.

6. The method according to claim 1, characterized in that The method of implementing non-rigid editing by using a single-view image editing model that has been trained to convergence and determining a target single-view image corresponding to the source image based on source text embedding data and target text embedding data corresponding to the source image for each source image comprises: Determine optimized target text embedding data based on the source text embedding data and the target text embedding data corresponding to the source image; The optimized target text embedding data corresponding to each source image is respectively input into the single-view image editing model that has been trained to convergence, and the single-view image editing model that has been trained to convergence is used to determine the corresponding target single-view image under each perspective, so as to realize non-rigid editing.

7. The method according to claim 6, characterized in that The step of determining optimized target text embedding data based on the source text embedding data and the target text embedding data corresponding to the source image includes: Decomposing the target text embedding data based on the source text embedding data corresponding to the source image to obtain a projected component and a vertical component, wherein the projected component represents a component of semantic commonality between the source text embedding data and the target text embedding data, and the vertical component represents a component of semantic difference between the source text embedding data and the target text embedding data; The optimized target text embedding data is determined based on the target text embedding data, the projection component and the vertical component.

8. The method according to claim 7, characterized in that The step of decomposing the target text embedding data based on the source text embedding data corresponding to the source image to obtain a projection component and a vertical component includes: The source text embedding data and the target text embedding data are input into a preset text embedding fusion algorithm to output the projection component and the vertical component.

9. The method according to claim 7, characterized in that: The step of determining the optimized target text embedding data based on the target text embedding data, the projection component and the vertical component comprises: The target text embedding data, the projection component and the vertical component are input into a preset text embedding optimization algorithm to output the optimized target text embedding data.

10. The method according to claim 1, characterized in that The single-view image editing model trained to convergence is a model in which a low-rank adaptation layer is added to the key vector and value vector of the cross-attention module of the single-view image generation model trained to convergence; Before the method of using the single-view image editing model that has been trained to convergence for each source image and determining the target single-view image corresponding to the source image based on the source text embedding data and the target text embedding data corresponding to the source image, the method further includes: Obtain a single-view image generation model that has been trained to convergence; Keeping the single-view image generation model that has been trained to convergence unchanged, training the training parameters of the low-rank adaptive layer until a first convergence condition is reached; The single-view image editing model that reaches the first convergence condition is determined as the single-view image editing model that has been trained to convergence.

11. The method according to claim 10, characterized in that The single-view image generation model that has been trained to convergence is a diffusion model, and the training process of the single-view image generation model that has been trained to convergence includes: Obtain an initial single-view image generation model; Keeping the source text embedding data corresponding to the source image unchanged, inputting the source text embedding data into the initial single-view image generation model, and training the training parameters of the initial single-view image generation model until a second convergence condition is reached; The single-view image generation model that reaches the second convergence condition is determined as the single-view image generation model that has been trained to convergence.

12. The method according to claim 11, characterized in that The step of optimizing the initial text embedding data to obtain each source text embedding data satisfying the optimization condition includes: Inputting the initial text embedding data into the initial single-view image generation model to output an image to be optimized; In response to the image to be optimized satisfying an optimization condition, determining that the initial text embedding data is source text embedding data; In response to the image to be optimized not meeting the optimization conditions, continue to adjust the initial text embedding data and keep the initial single-view image generation model unchanged until the output image to be optimized meets the optimization conditions, and determine the text embedding data corresponding to the optimization conditions when the optimization conditions are met as the source text embedding data.

13. An editing device, characterized in that: include: Memory and processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method according to any one of claims 1 to 12.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 12 when executed by a processor.

15. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 12 when being executed by a processor.

Images