Picture generation method and apparatus, storage medium, and device

Abstract

This disclosure provides an image generation method, apparatus, storage medium, and device. By obtaining a first view of an object, extracting the object's three-dimensional structural features and appearance features from the first view, generating a first three-dimensional model of the object based on the object's three-dimensional structural features and appearance features, and generating a second view with a different perspective from the first view based on the object's first three-dimensional model, it is beneficial to improve the generation efficiency and accuracy of multi-viewpoints, reduce labor costs, and shorten the design cycle.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to an image generation method and apparatus, storage medium and device. Background Technology

[0002] In the appearance and structural design of products / equipment, multi-angle views are often relied upon. Related technologies primarily involve manual modeling, hand-drawn rendering, or 3D design software to generate drawings. However, this approach suffers from long design cycles, high costs, and cumbersome modification and iteration processes. Therefore, how to efficiently and accurately generate multi-angle views of products / equipment is a technical problem that needs to be solved. Summary of the Invention

[0003] The purpose of this disclosure is to provide an image generation method, apparatus, storage medium, and device to efficiently and accurately generate multi-angle views of a product / equipment.

[0004] Other features and advantages of this disclosure will become apparent from the following detailed description, or may be learned in part from practice of this disclosure.

[0005] According to a first aspect of this disclosure, an image generation method is provided, the image generation method comprising: obtaining a first view of an object; extracting three-dimensional structural features and appearance features of the object from the first view; generating a first three-dimensional model of the object based on the three-dimensional structural features and the appearance features; and generating a second view of the object based on the first three-dimensional model of the object, wherein the first view and the second view have different perspectives.

[0006] In some exemplary embodiments of this disclosure, obtaining the first view of the object includes: obtaining descriptive text of the object and information about its appearance features; inputting the descriptive text into a first image generation model and generating a third view containing the object through the first image generation model; inputting the information about the appearance features and the third view into a second image generation model and generating a first view containing the object and its appearance features through the second image generation model.

[0007] In some exemplary embodiments of this disclosure, the step of extracting the three-dimensional structural features and appearance features of the object from the first view includes: inputting the first view into a feature extraction network, extracting the appearance features of the object through the feature extraction network; and performing depth estimation on the pixels in the first view to obtain the three-dimensional structural features of the object.

[0008] In some exemplary embodiments of this disclosure, generating a first three-dimensional model of the object based on the three-dimensional structural features and the appearance features includes: performing three-dimensional reconstruction of the object based on the three-dimensional structural features to obtain a second three-dimensional model of the object; and performing feature fusion on the second three-dimensional model and the appearance features to obtain the first three-dimensional model.

[0009] In some exemplary embodiments of this disclosure, the method further includes: determining the degree of consistency of the structure and appearance of the object in the first view and the second view based on a preset evaluation model; and outputting the second view in response to the degree of consistency being higher than a preset threshold.

[0010] According to a second aspect of this disclosure, an image generation apparatus is provided, comprising: The Get module is used to retrieve the first view of an object.

[0011] An extraction module is used to extract the three-dimensional structural features and appearance features of the object from the first view.

[0012] The first generation module is used to generate a first three-dimensional model of the object based on the three-dimensional structural features and the appearance features.

[0013] The second generation module is used to generate a second view of the object based on the first three-dimensional model of the object, wherein the first view and the second view have different perspectives.

[0014] In some exemplary embodiments of this disclosure, the acquisition module is configured to: acquire descriptive text of the object and information about the appearance features; input the descriptive text into a first image generation model and generate a third view containing the object through the first image generation model; input the information about the appearance features and the third view into a second image generation model and generate a first view containing the object and the appearance features through the second image generation model.

[0015] In some exemplary embodiments of this disclosure, the extraction module is configured to: input the first view into a feature extraction network, extract the appearance features of the object through the feature extraction network; and perform depth estimation on the pixels in the first view to obtain the three-dimensional structural features of the object.

[0016] In some exemplary embodiments of this disclosure, the first generation module is configured to: perform three-dimensional reconstruction of the object based on the three-dimensional structural features to obtain a second three-dimensional model of the object; and perform feature fusion of the second three-dimensional model and the appearance features to obtain the first three-dimensional model.

[0017] In some exemplary embodiments of this disclosure, the apparatus further includes: The determination module is used to determine the degree of consistency of the structure and appearance of the object in the first view and the second view based on a preset evaluation model.

[0018] The output module is used to output the second view in response to the consistency level being higher than a preset threshold.

[0019] According to a third aspect of this disclosure, a terminal device is provided, including a processor and a memory, the memory being used to store executable instructions of the processor; wherein the processor is configured to perform the method of the first aspect by executing the executable instructions.

[0020] According to a fourth aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method of the first aspect described above.

[0021] The image generation method, apparatus, storage medium, and device provided in this disclosure obtain a first view of an object, extract the three-dimensional structural features and appearance features of the object from the first view, generate a first three-dimensional model of the object based on the three-dimensional structural features and appearance features, and generate a second view with a different perspective from the first view based on the first three-dimensional model of the object. This is beneficial to improving the generation efficiency and accuracy of multi-viewpoints, reducing labor costs, and shortening the design cycle.

[0022] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0023] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0024] Figure 1 A flowchart of an image generation method according to an embodiment of this disclosure is shown; Figure 2 This is a flowchart of a view generation method provided in an embodiment of this disclosure; Figure 3 This is a schematic diagram of a method for generating text images according to an embodiment of this disclosure; Figure 4 This is a schematic diagram of another view generation method provided in an embodiment of this disclosure; Figure 5This is a schematic diagram of a feature extraction method provided in an embodiment of this disclosure; Figure 6 This is a flowchart of a three-dimensional model generation method provided in an embodiment of this disclosure; Figure 7 This is a flowchart of a consistency identification method provided in an embodiment of this disclosure; Figure 8 This is a flowchart of an image generation method provided in an embodiment of this disclosure; Figure 9 This is a schematic diagram of yet another view generation method provided in this disclosure embodiment; Figure 10 This diagram illustrates an image generation apparatus according to an embodiment of the present disclosure; Figure 11 A structural block diagram of a terminal device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0025] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0026] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0027] With the development of generative artificial intelligence technology, deep learning-based image generation methods have begun to be applied to industrial design. However, this method still has many shortcomings. For example, in locomotive design, it is often necessary to generate multi-view views of the locomotive, such as front view, side view, and top view. However, multi-view views generated based on artificial intelligence models often suffer from poor consistency issues, such as misalignment of the overall vehicle outline, asymmetry of key structures, and disordered perspective relationships, rendering the generated multi-view views unusable.

[0028] To address the aforementioned issues, this disclosure provides an image generation method, apparatus, storage medium, and device to efficiently and accurately generate multi-angle views of products / equipment, thus resolving the problem of poor consistency across multiple viewpoints in the prior art.

[0029] The following examples illustrate the solutions provided in this disclosure.

[0030] Figure 1 A flowchart of an image generation method according to an embodiment of this disclosure is shown. This method can be executed by a terminal device, as exemplarily understood. The terminal device can be understood as any device with image processing and generation capabilities, such as a laptop, desktop computer, or smartphone, but is not limited to the devices listed herein. Figure 1 As shown, in some embodiments, the image generation method provided by this disclosure may include the following steps.

[0031] In step S101, the first view of the object is obtained.

[0032] In this disclosure, the term "object" can be understood as any kind of equipment or product. For ease of understanding, it can be exemplarily understood as a locomotive in this disclosure.

[0033] The first view can be understood as a view of an object from a certain perspective, such as the front view, side view, or top view.

[0034] In some examples, the first view can be obtained from a pre-set storage device or server.

[0035] In other examples, the first device can be obtained through a user interface. For instance, the user uploads the first view to the user interface using the upload function provided on the user interface, and then obtains the first view from the user interface through the interface and data channel between the user and the user interface.

[0036] In other examples, the first view can also be generated using an artificial intelligence model. For example, Figure 2 This is a flowchart of a view generation method provided in an embodiment of this disclosure. For example... Figure 2 As shown, in some examples, the first view can be generated using the methods in steps S201-S205.

[0037] In step S201, the description text of the object and information on its appearance features are obtained.

[0038] The object description text is used to describe the object's type (e.g., locomotive, mobile phone, computer), its structure, the relative positions and combinations of the structures.

[0039] In some examples, the appearance characteristics referred to in this disclosure include text used to describe the appearance of an object. For example, the object's color, style, material, etc.

[0040] In other examples, information about appearance features may include a style map, which includes information such as color, style, and material.

[0041] In step S203, the descriptive text is input into the first image generation model, and a third view containing the object is generated through the first image generation model.

[0042] In some examples, the first image generation model referred to in this disclosure embodiment can be any text-to-image model. The first image generation model can generate an image that matches the content described in the descriptive text.

[0043] Take conditional generative adversarial networks as an example. Figure 3 This is a schematic diagram of a method for generating text-to-images according to an embodiment of this disclosure. For example... Figure 3 As shown, in some examples, a conditional generative adversarial network (GAN) may include a generator and a discriminator. The generator generates an image containing the object based on its descriptive text and sends the image to the discriminator. The discriminator determines the degree of match between the generated image and the descriptive text. If the match is less than a preset threshold, the generator is triggered to regenerate the image, and the regenerated image is then compared to the descriptive text. This process continues until the match is greater than or equal to the preset threshold, at which point a third-party view is obtained and output.

[0044] In step S205, the information on the appearance features and the third view are input into the second image generation model, and a first view containing the object and its appearance features is generated through the second image generation model.

[0045] In this embodiment of the disclosure, the third view generated based on the object's descriptive text includes the object's structure but excludes the object's appearance features such as color and texture.

[0046] In this embodiment of the disclosure, appearance features are added to a third view using a second image generation model to obtain a first view. At this time, the objects contained in the first view have appearance features such as color and texture.

[0047] The second image model can be any type of model, such as a neural network model or a machine learning model. The input to the second image generation model includes information about appearance features (such as descriptive text or style maps of the appearance features) and a third view. The output is a first view that incorporates the appearance features.

[0048] for example, Figure 4 This is a schematic diagram of another view generation method provided in an embodiment of this disclosure. For example... Figure 4 As shown, in some examples, the first view can be generated using the following method.

[0049] In step S401, information on the appearance features input by the user is received.

[0050] In step S403, it is determined whether the appearance feature information input by the user meets the requirements.

[0051] For example, whether the input information format meets the preset format requirements. Whether the value range of the set parameters such as color and brightness is within the preset value range, etc. If it does not meet the requirements, proceed to step S405; otherwise, proceed to step S407.

[0052] In step S405, the user is prompted to re-enter the information, and the process jumps to step S401.

[0053] In step S407, the second image generation model is invoked, and the received appearance feature information and the third view are input into the second image generation model to generate the first view.

[0054] This disclosure enables the rapid addition of appearance features to a third view using a second image generation model, thereby improving the generation efficiency of the first view.

[0055] In step S103, the three-dimensional structural features and appearance features of the object are extracted from the first view.

[0056] In some examples, the three-dimensional structural features (such as the structures contained in the object, the shape and size of the structures, and the relative positional relationships between the structures) and appearance features (such as color, texture, brightness, transparency, etc.) of the object can be extracted from the first view through a pre-set artificial intelligence model.

[0057] for example, Figure 5 This is a schematic diagram of a feature extraction method provided in an embodiment of this disclosure. Figure 5 As shown, a first view can be input into a preset artificial intelligence model, which simultaneously outputs the object's 3D structural and appearance features. The artificial intelligence model from which the user extracts the 3D structural and appearance features can be any type of artificial intelligence model, such as a neural network model or a machine learning model. The training method for this artificial intelligence model can be found in related technical documentation on model training methods, and will not be elaborated upon here.

[0058] It should be noted that the methods in the above examples are merely exemplary and not the only limitations on methods for extracting 3D structural and appearance features. In fact, all methods that can be used to extract 3D structural and appearance features from images can be applied in this disclosure. For example, in some examples, a first view can be input into a preset feature extraction network (such as a convolutional neural network) to extract the object's appearance features from the first view. The 3D structural features of the object are obtained by performing depth estimation on the pixels in the first view. Methods for depth estimation of pixels in the view can be found in related technologies and will not be elaborated here.

[0059] In step S105, a first three-dimensional model of the object is generated based on the object's three-dimensional structural features and appearance features.

[0060] For example, in some implementations, the three-dimensional structural features and appearance features of an object can be input into a preset neural implicit field network. The neural implicit field network is used to reconstruct a three-dimensional mesh model of the object and associate the appearance features with the three-dimensional mesh model, outputting a first three-dimensional model. The method for reconstructing the three-dimensional mesh model of the object using a neural implicit field network and associating the appearance features with the three-dimensional mesh model can be found in related technologies and will not be elaborated here.

[0061] For example, Figure 6 This is a flowchart of a three-dimensional model generation method provided in an embodiment of this disclosure. For example... Figure 6 As shown, in some examples, the first three-dimensional model of an object can be generated using the following method.

[0062] In step S601, the object is reconstructed in three dimensions based on its three-dimensional structural features to obtain a second three-dimensional model of the object.

[0063] For example, in some examples, the three-dimensional structural features of an object can be extracted based on a first view. These features are encoded as spatial geometric constraints, including but not limited to the spatial coordinates of the object's surface, constraints on the division of the object's internal and external regions, normal constraints at each point on the object's surface, and prior geometric information. Subsequently, the extracted three-dimensional structural features are input into a pre-defined implicit representation network. This network can be any of OccNet, DeepSDF, or ConvOccNet. Based on the input three-dimensional structural features, the network learns the signed distance field (SDF) from any coordinate point in three-dimensional space to the object's surface, or the occupancy probability field indicating that any coordinate point in three-dimensional space belongs to the object's interior, through deep learning training, thus achieving an implicit representation of the object's three-dimensional geometric structure. Next, an isosurface extraction algorithm is used to process the signed distance field or occupancy probability field output by the implicit representation network. The preferred isosurface extraction algorithm is the Marching Cubes algorithm. This algorithm extracts the complete surface of the object from the signed distance field or occupancy probability field, reconstructing the object's three-dimensional geometric contour. Finally, the extracted object surface is smoothed and topologically optimized to generate a smooth, topologically natural 3D mesh model, i.e., the second 3D model. The methods for smoothing and topologically optimizing the extracted surface, as well as the method for extracting the object surface using the same isosurface extraction algorithm as the Forbidden City, can be found in relevant technical documents and will not be elaborated upon here.

[0064] For example, in other examples, depth information of the object can be extracted based on the first view using depth estimation algorithms or stereo matching algorithms. Then, the object's 3D structural features can be extracted from the depth information. These 3D structural features include, but are not limited to, the 3D coordinates of each spatial point on the object's surface, the normal vectors corresponding to each spatial point, the local curvature of the object's surface, the object's contour constraints, and spatial topological relationships. Subsequently, the extracted 3D structural features are converted to generate a 3D point cloud corresponding to the object. The 3D point cloud is a sparse set of points representing the object's 3D geometry. Next, the generated sparse point cloud is preprocessed, sequentially performing point cloud denoising, point cloud registration, and point cloud completion operations. This removes noise points from the point cloud, corrects point cloud offset deviations, and fills in missing regions, resulting in a dense and regular 3D point cloud that ensures the point cloud can completely represent the object's 3D geometry. Finally, a surface fitting algorithm is used to process the preprocessed dense point cloud. The surface fitting algorithm can be any one of the Poisson reconstruction algorithm, moving least squares (MLS) algorithm, or Alpha Shapes algorithm. The discrete point cloud is transformed into a continuous closed triangular mesh model through the surface fitting algorithm, and the three-dimensional reconstruction of the object is completed to obtain the second three-dimensional model of the object.

[0065] In step S603, feature fusion is performed on the second three-dimensional model and the appearance features to obtain the first three-dimensional model.

[0066] In some examples, the second 3D model and appearance features can be encoded separately to obtain a first feature vector corresponding to the second 3D model and a second feature vector corresponding to the appearance features. The first and second feature vectors are concatenated to obtain a third feature vector. This third feature vector is then input into a preset mesh texture mapping network. This network maps the appearance features to the vertices and faces of the second 3D model using the geometric constraints contained in the third feature vector, thus obtaining the first 3D model.

[0067] In other examples, the first and second feature vectors can be input into a pre-defined self-attention model. The self-attention model uncovers the intrinsic relationship between the first and second feature vectors, and based on this relationship, the first and second feature vectors are fused to obtain a fused vector. This fused vector is then input into a neural implicit field generation network, which generates an implicit field. This implicit field represents the mapping relationship between 3D spatial coordinates and structural and appearance features. An isosurface extraction algorithm is used to extract the object's surface from the implicit field, and the appearance features are mapped onto the surface to generate the first 3D model. The method for extracting the object's surface from the implicit field using the isosurface extraction algorithm can be found in related technologies and will not be elaborated upon here.

[0068] This disclosure reconstructs a second three-dimensional model of the object based on its three-dimensional structural features (excluding appearance features such as color, texture, and brightness). By fusing the second three-dimensional model and appearance features, it is beneficial to quickly obtain a three-dimensional model that includes appearance features, thereby improving the generation efficiency of other viewpoints and ensuring the uniformity of appearance in multiple viewpoints.

[0069] In step S107, a second view of the object is generated based on the first three-dimensional model of the object, and the first view and the second view have different perspectives.

[0070] In some implementations, the first 3D model can be projected onto a preset 2D plane to generate a second view. The steps are as follows: In S1, obtain the projection parameters of the object's first 3D model and first view.

[0071] The first three-dimensional model contains complete geometric structure and appearance features. The projection parameters include the projection type (orthographic projection or perspective projection) corresponding to the first view, the projection matrix, the projection direction, and the two-dimensional imaging plane parameters.

[0072] In S2, determine the target projection angle of the second view.

[0073] For example, based on the projection parameters of the first view, the projection direction and projection matrix can be adjusted to change the projection angle, so that the projection direction of the second view forms a preset angle with the projection direction of the first view, ensuring that there is a difference in the viewing angle between the first view and the second view. For example, if the first view is a front view (projection direction perpendicular to the front of the object), then the second view can be set as a side view (projection direction perpendicular to the side of the object) or a top view (projection direction perpendicular to the top of the object).

[0074] In S3, the coordinates of all vertices of the first 3D model are mapped to the 2D imaging plane through the adjusted projection matrix to obtain the coordinates of each vertex on the 2D plane; at the same time, the appearance features such as texture and color of the first 3D model are synchronously mapped to the pixels corresponding to the 2D coordinates to ensure the consistency between appearance details and geometric structure, and generate the initial 2D view.

[0075] In S4, the initial two-dimensional view is optimized, including operations such as pixel filling, edge smoothing, and texture alignment. This repairs problems such as missing pixels and jagged edges that occur during projection, making the geometric outline of the second view clear and the appearance realistic. Finally, a second view with a different perspective from the first view is obtained.

[0076] according to Figures 1-6 In this embodiment, the present disclosure obtains a first view of an object, extracts the object's three-dimensional structural features and appearance features from the first view, generates a first three-dimensional model of the object based on the object's three-dimensional structural features and appearance features, and generates a second view with a different perspective from the first view based on the object's first three-dimensional model. This is beneficial to improve the generation efficiency and accuracy of multi-viewpoints, reduce labor costs, and shorten the design cycle.

[0077] Figure 7 This is a flowchart of a consistency identification method provided in an embodiment of this disclosure. Figure 7 As shown, in some implementations, the degree of consistency of the structure and appearance of an object in the first and second views can be determined by the following method.

[0078] In step S701, based on a preset evaluation model, the degree of consistency between the structure and appearance of the object in the first view and the second view is determined.

[0079] The evaluation model described in this disclosure is used to evaluate the consistency of the structure and appearance of the same object across multiple views. The input to the evaluation model consists of multiple views containing the same object, such as the first view and the second view as described in this disclosure. The output is a value indicating the consistency of the object's structure and appearance across these views. The evaluation model can be trained using any model, such as a neural network model or a machine learning model. Training methods for the evaluation model can be found in related art, and will not be elaborated upon here.

[0080] In step S703, in response to the consistency level being higher than a preset threshold, a second view is output.

[0081] In this embodiment of the disclosure, by evaluating the model, the degree of consistency between the structure and appearance of the object in the first view and the second view is determined, and when the degree of consistency is higher than a preset threshold, the second view is output, which can ensure the consistency between the structure and appearance of the object in the first view and the second view.

[0082] Example, Figure 8 This is a flowchart of an image generation method provided in an embodiment of this disclosure. For example... Figure 8 As shown, in some examples, a second view can be generated and output using the following method.

[0083] In step S801, the first view is obtained.

[0084] In step S803, the first view is input into a preset neural network, and the second view is generated through the neural network.

[0085] In this embodiment, a preset neural network can be used to execute the method for generating the second view in the above embodiments. For example, Figure 9 This is a schematic diagram of yet another view generation method provided in this disclosure. For example... Figure 9 As shown, in some examples, the pre-defined neural network may include a feature extraction network, a 3D reconstruction network, and a mapping network. The feature extraction network is used to extract the appearance features of the object from the first view. The 3D reconstruction network is used to extract 3D structural features from the first view and generate a first 3D model based on the 3D reconstruction features and appearance features. The mapping network is used to map the first 3D model onto a 2D plane to obtain a second view. Of course, Figure 9 This is merely an illustrative example of the neural networks referred to in this disclosure, and not a unique limitation.

[0086] In step S805, structural features and appearance features are extracted from the first view and the second view.

[0087] In step S807, a first three-dimensional model is reconstructed based on the structural and appearance features extracted from the first view, and a fourth three-dimensional model is reconstructed based on the structural and appearance features extracted from the second view.

[0088] In step S809, the similarity between the first three-dimensional model and the fourth three-dimensional model is calculated.

[0089] The similarity between the first 3D model and the fourth 3D model can be calculated based on the similarity calculation method provided by existing technology, which will not be elaborated here.

[0090] In step S811, when the similarity between the first 3D model and the fourth 3D model is greater than or equal to a preset similarity, the second view is output.

[0091] Figure 8 For the implementation method and beneficial effects of the embodiments, please refer to Figures 1-7 The execution method and beneficial effects of any of the embodiments are not described in detail here.

[0092] Figure 10 This diagram illustrates an image generation apparatus according to an embodiment of the present disclosure. Figure 10 As shown, in some embodiments, the image generation apparatus 900 may include: Module 901 is used to obtain the first view of an object.

[0093] Extraction module 902 is used to extract the three-dimensional structural features and appearance features of the object from the first view.

[0094] The first generation module 903 is used to generate a first three-dimensional model of the object based on the three-dimensional structural features and the appearance features.

[0095] The second generation module 904 is used to generate a second view of the object based on the first three-dimensional model of the object, wherein the first view and the second view have different perspectives.

[0096] In some exemplary embodiments of this disclosure, the acquisition module 901 is configured to: acquire descriptive text of the object and information about the appearance features; input the descriptive text into a first image generation model and generate a third view containing the object through the first image generation model; input the information about the appearance features and the third view into a second image generation model and generate a first view containing the object and the appearance features through the second image generation model.

[0097] In some exemplary embodiments of this disclosure, the extraction module 902 is configured to: input the first view into a feature extraction network, extract the appearance features of the object through the feature extraction network; and perform depth estimation on the pixels in the first view to obtain the three-dimensional structural features of the object.

[0098] In some exemplary embodiments of this disclosure, the first generation module 903 is configured to: perform three-dimensional reconstruction of the object based on the three-dimensional structural features to obtain a second three-dimensional model of the object; and perform feature fusion of the second three-dimensional model and the appearance features to obtain the first three-dimensional model.

[0099] In some exemplary embodiments of this disclosure, the apparatus further includes: The determination module is used to determine the degree of consistency of the structure and appearance of the object in the first view and the second view based on a preset evaluation model.

[0100] The output module is used to output the second view in response to the consistency level being higher than a preset threshold.

[0101] Figure 10 The execution method and beneficial effects of the image generation device in the embodiment can be found in [reference needed]. Figures 1-9 Any of the embodiments described herein will not be repeated here.

[0102] In some embodiments, this disclosure also provides a terminal device including a processor and a memory, the memory being used to store executable instructions of the processor; wherein the processor is configured to perform the method in any of the above method embodiments by executing the executable instructions.

[0103] Figure 11 A structural block diagram of a terminal device according to an embodiment of this disclosure is shown. Referring below... Figure 11 To describe a terminal device 1000 according to this embodiment of the present invention. Figure 11 The terminal device 1000 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0104] like Figure 11 As shown, the terminal device 1000 is presented in the form of a general-purpose computing device. The components of the terminal device 1000 may include, but are not limited to: at least one processing unit 1010 (included in one or more processors), at least one storage unit 1020 (included in one or more memories), and a bus 1030 connecting different system components (including storage unit 1020 and processing unit 1010).

[0105] The storage unit stores program code that can be executed by the processing unit 1010, causing the processing unit 1010 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of the present invention.

[0106] Storage unit 1020 may include readable media in the form of volatile storage units, such as random access memory (RAM) 1021 and / or cache memory 1022, and may further include read-only memory (ROM) 1023.

[0107] Storage unit 1020 may also include a program / utility 1024 having a set (at least one) program module 1025, such program module 1025 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0108] Bus 1030 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the multiple bus structures.

[0109] Terminal device 1000 can also communicate with one or more external devices 1040 (e.g., keyboard, pointing device, Bluetooth device, etc.), one or more devices that enable a user to interact with terminal device 1000, and / or any device that enables terminal device 1000 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 1050. Furthermore, terminal device 1000 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 1060. As shown, network adapter 1060 communicates with other modules of terminal device 1000 via bus 1030. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with terminal device 1000, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0110] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0111] In exemplary embodiments of this disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the methods described above is stored. In some possible embodiments, various aspects of the invention may also be implemented as a program product comprising program code that, when the program product is run on a terminal device, causes the terminal device to perform the steps of the various exemplary embodiments of the invention described in the "Exemplary Methods" section of this specification.

[0112] A program product for implementing the above-described method according to embodiments of the present invention is described. This product may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, the readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0113] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0114] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0115] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0116] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0117] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0118] Furthermore, although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0119] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the appended claims.

Claims

1. A picture generation method, characterized by, include: Get the first view of the object; Extract the three-dimensional structural features and appearance features of the object from the first view; Based on the three-dimensional structural features and the appearance features, a first three-dimensional model of the object is generated; Based on the first 3D model of the object, a second view of the object is generated, wherein the first view and the second view have different perspectives.

2. The method of claim 1, wherein, The process of obtaining the first view of the object includes: Obtain the description text of the object, as well as information about its appearance features; The descriptive text is input into a first image generation model, and a third view containing the object is generated by the first image generation model. The information of the appearance features and the third view are input into the second image generation model, and a first view containing the object and the appearance features is generated through the second image generation model.

3. The method of claim 1, wherein, The step of extracting the three-dimensional structural features and appearance features of the object from the first view includes: The first view is input into a feature extraction network, and the appearance features of the object are extracted by the feature extraction network. Depth estimation is performed on the pixels in the first view to obtain the three-dimensional structural features of the object.

4. The method of claim 1, wherein, The step of generating a first three-dimensional model of the object based on the three-dimensional structural features and the appearance features includes: Based on the three-dimensional structural features, the object is reconstructed in three dimensions to obtain a second three-dimensional model of the object; The second 3D model and the appearance features are fused to obtain the first 3D model.

5. The method according to any one of claims 1-4, characterized in that, The method further includes: Based on a preset evaluation model, the degree of consistency between the structure and appearance of the object in the first view and the second view is determined; In response to the consistency level being higher than a preset threshold, the second view is output.

6. An image generating apparatus characterized by comprising: include: The get module is used to get the first view of an object; The extraction module is used to extract the three-dimensional structural features and appearance features of the object from the first view; The first generation module is used to generate a first three-dimensional model of the object based on the three-dimensional structural features and the appearance features; The second generation module is used to generate a second view of the object based on the first three-dimensional model of the object, wherein the first view and the second view have different perspectives.

7. The apparatus of claim 6, wherein, The acquisition module is used for: Obtain the descriptive text of the object, as well as information about its appearance features; The descriptive text is input into a first image generation model, and a third view containing the object is generated by the first image generation model. The information of the appearance features and the third view are input into the second image generation model, and a first view containing the object and the appearance features is generated through the second image generation model.

8. The apparatus of claim 6, wherein, The extraction module is used for: The first view is input into a feature extraction network, and the appearance features of the object are extracted by the feature extraction network. Depth estimation is performed on the pixels in the first view to obtain the three-dimensional structural features of the object.

9. The apparatus of claim 6, wherein, The first generation module is used for: Based on the three-dimensional structural features, the object is reconstructed in three dimensions to obtain a second three-dimensional model of the object; The second 3D model and the appearance features are fused to obtain the first 3D model.

10. The apparatus of any one of claims 6-9, wherein, The device further includes: The determination module is used to determine the degree of consistency of the structure and appearance of the object in the first view and the second view based on a preset evaluation model. The output module is used to output the second view in response to the consistency level being higher than a preset threshold.

11. A terminal device, comprising: include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the method of any one of claims 1 to 5 by executing the executable instructions.

12. A computer readable storage medium having stored thereon a computer program, characterized in that, When the computer program is executed by the processor, it implements the method described in any one of claims 1 to 5.

Images