Three-view generation method, system, equipment and medium

Abstract

The invention is suitable for the technical field of computers, and provides a three-view generation method which comprises the following steps: inputting a text prompt into a text encoder to obtain a text feature, and inputting a reference view into an image encoder to obtain a first image feature; inputting the first image feature into a self-attention module to obtain first attention; adding the first attention and the first image feature to obtain a second image feature; inputting the text feature and the second image feature into a cross-attention module to obtain second attention; adding the second attention, the text feature and the second image feature to obtain a third image feature; and inputting the third image feature into a feedback module to obtain a three-view image. According to the three-view generation method, the accuracy of detail information and texture information of the three views and the correlation and correctness of the three views can be ensured.

Description

Technical Field

[0001] The present application belongs to the field of computer technology, and in particular, relates to a three-view generating method, system, device and medium. Background Art

[0002] With the rapid development of computer technology, 3D modeling and rendering technology has become an indispensable part of digital content creation. In many fields such as product design, architectural design, and film and television special effects, accurate and detailed three-view display is crucial to improving user experience and decision-making efficiency. Traditional methods mostly rely on manual modeling or rule-based automatic generation, which is not only time-consuming and labor-intensive, but also difficult to ensure consistency and detail richness between views. In recent years, deep learning, especially the Diffusion model, has made significant progress in the field of image generation, but when it is directly applied to complex three-view generation, there are still problems such as blurred details and inconsistency between views.

[0003] In the prior art, the construction of three-dimensional images usually includes the following three methods: (1) Convolutional Auto Encoder Extreme Learning Machine (CAE-ELM) method, which combines the characteristics of convolutional neural network (CNN), auto encoder (Auto Encoder) and extreme learning machine (ELM), and can simultaneously extract two different three-views of the three-dimensional image, and optimize the classification performance by combining the two three-views. The above-mentioned convolutional auto encoder extreme learning machine method has the advantages of fast training speed, small number of connections and rotation invariance. (2) Multi-core learning method, this method constructs different base kernels for different views, and finds the structural association between different views in a linear or nonlinear way, effectively fuses multi-view information to obtain a consistent kernel to improve clustering performance. (3) Multi-view clustering algorithm based on graph model, this method transforms the clustering problem into a graph segmentation problem by constructing graph structures under different views, and then learning a consistent graph that fuses the topological structures of all views.

[0004] Among the above three methods, method (I) uses a method based on convolutional neural networks or generative adversarial networks to directly generate images from different perspectives. It is difficult to capture and accurately present the subtle differences of complex objects from different perspectives, resulting in the generated views being not rich enough in details. In addition, there is a lack of effective mechanisms to ensure the correlation and correctness between different views, which affects the final three-dimensional display effect. Method (II) How to choose a suitable kernel function and how to effectively combine multiple kernel functions are two key issues. The selection of a suitable kernel function often depends on the researcher's experience and intuitive understanding of the data. This subjectivity leads to the selection of a non-optimal kernel function in some cases, thus affecting the clustering effect. The interpretability of the clustering results in method (III) depends on the initialization of the graph, and the quality of the initial graph is usually difficult to guarantee. Summary of the invention

[0005] The embodiments of the present application provide a three-view generating method, system, device and medium, which can solve one of the above-mentioned problems in the prior art.

[0006] In a first aspect, an embodiment of the present application provides a three-view generating method, including:

[0007] Inputting the text prompt into a text encoder to obtain a text feature, and inputting the reference view into an image encoder to obtain a first image feature;

[0008] Inputting the first image feature into a self-attention module to obtain a first attention;

[0009] Adding the first attention to the first image feature to obtain a second image feature;

[0010] Inputting the text feature and the second image feature into a cross-attention module to obtain a second attention;

[0011] Adding the second attention, the text feature and the second image feature to obtain a third image feature;

[0012] The third image feature is input into a feedback module to obtain a three-view image.

[0013] Furthermore, the self-attention module includes a first normalization layer and a self-attention layer;

[0014] The step of inputting the first image feature into a self-attention module to obtain a first attention comprises:

[0015] Inputting the first image feature into the first normalization layer to obtain a first standard feature;

[0016] Adding the first image feature and the first standard feature to obtain a self-attention layer input feature;

[0017] The input feature of the self-attention layer is input into the self-attention layer to perform self-attention calculation to obtain the first attention.

[0018] Furthermore, the step of inputting the self-attention layer input feature into the self-attention layer for self-attention calculation to obtain a first attention comprises:

[0019] Learning the detail features of the input features of the self-attention layer through the detail information matrix, and learning the texture features of the input features of the self-attention layer through the texture feature matrix;

[0020] The calculation formula of the first attention is as follows:

[0021]

[0022] Among them, At1 represents the first attention, I1 represents the input feature of the self-attention layer, W represents the texture feature matrix, X represents the detail information matrix, and * represents bitwise multiplication.

[0023] Furthermore, the cross-attention module includes a second normalization layer and a cross-attention layer;

[0024] The step of inputting the text feature and the second image feature into a cross-attention module to obtain a second attention comprises:

[0025] Inputting the second image feature into the second normalization layer to obtain a second standard feature;

[0026] Perform feature correlation calculation on the second standard feature and the text feature to obtain a cross-attention layer input feature;

[0027] The cross-attention layer input feature is input into the cross-attention layer to perform cross-attention calculation to obtain a second attention.

[0028] Furthermore, the calculation formula for obtaining the cross-attention layer input features is as follows:

[0029]

[0030] Among them, I2 represents the second image feature, I3 represents the text feature, I4 represents the cross-attention layer input feature, M1 and M2 represent learnable mask matrices, and * represents bitwise multiplication.

[0031] Furthermore, performing feature correlation calculation on the second standard feature and the text feature to obtain a cross-attention layer input feature includes:

[0032] The correlation between the text feature and the detail feature of the second standard feature is learned through a detail correlation matrix, and the correlation between the text feature and the texture feature of the second standard feature is learned through a texture correlation matrix;

[0033] The calculation formula of the second attention is as follows:

[0034]

[0035] Among them, At2 represents the second attention, I4 represents the cross-attention layer input feature, Y represents the detail correlation matrix, T represents the texture correlation matrix, M3 represents the learnable mask matrix, and * represents bitwise multiplication.

[0036] Further, the feedback module includes a third normalization layer and a feedback layer;

[0037] The step of inputting the third image feature into a feedback module to obtain a three-view image comprises:

[0038] Inputting the third image feature into the third normalization layer to obtain a third standard feature;

[0039] The third standard feature is input into the feedback layer to obtain a three-view image.

[0040] In a second aspect, an embodiment of the present application provides a three-view generation system, including:

[0041] A first processing module: used for inputting the text prompt into a text encoder to obtain text features, and inputting the reference view into an image encoder to obtain first image features;

[0042] A second processing module: used for inputting the first image feature into a self-attention module to obtain a first attention;

[0043] A third processing module: configured to add the first attention to the first image feature to obtain a second image feature;

[0044] A fourth processing module: used for inputting the text feature and the second image feature into a cross-attention module to obtain a second attention;

[0045] A fifth processing module: configured to add the second attention feature, the text feature and the second image feature to obtain a third image feature;

[0046] The sixth processing module is used to input the third image feature into the feedback module to obtain a three-view image.

[0047] In a third aspect, an embodiment of the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned three-view generation method when executing the computer program.

[0048] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, including a computer-readable storage medium storing a computer program, and the computer program implements the above-mentioned three-view generating method when executed by a processor.

[0049] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0050] The present application discloses a three-view generation method, which sets a three-view generation module and designs a related algorithm so that the three-view generation module can be combined with any generation model for training, so that the finally trained three-view generation model can generate other views with reference to the reference view and text prompt provided by the user, and repeat the process to continuously refine the details. In addition, the present application introduces a detail information matrix and a texture feature matrix to ensure that the three-view generation model can learn the control of the reference view on the detail information and texture information during the training process, thereby ensuring the accuracy of the detail information and texture information of the three views. In addition, the detail correlation matrix and the texture correlation matrix are introduced to ensure the similarity between the image features and the text features, and further ensure the correlation and correctness of the three views. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0052] Figure 1 It is a flowchart of a three-view image generation method provided by an embodiment of the present invention;

[0053] Figure 2 It is a structural schematic diagram of a three-view image generation system provided by an embodiment of the present invention;

[0054] Figure 3 It is a structural diagram of a computer device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0055] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0056] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0057] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0058] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0059] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0060] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0061] See also Figure 1 As shown, the present invention is a three-view generating method, comprising the following steps:

[0062] S100, inputting a text prompt into a text encoder to obtain a text feature, and inputting a reference view into an image encoder to obtain a first image feature;

[0063] In this embodiment, when generating three views, the user inputs text prompts for the three views, such as "front view of a cylinder", "side view of a rectangular block", etc., and then inputs the text prompts into a text encoder to convert the text prompts into word embeddings to obtain text features.

[0064] In this embodiment, by allowing the user to input specific text prompts to specify the required three-view type (such as front view, side view, etc.) and object shape (such as cylinder, cuboid, etc.), it is possible to generate views that better meet user needs, thereby improving the flexibility of the three-view generation method. In addition, the input of text prompts lowers the technical threshold, so that non-professional users can also easily operate and obtain the required three-view. Users no longer need to have an in-depth understanding of complex graphics processing or programming knowledge, and can achieve their goals by simply entering descriptive text, thereby improving the user experience. In addition, the text encoder converts the text prompts into word embeddings to obtain text features, which can more accurately control the generation process, reduce errors and unnecessary iterations, and thus improve the efficiency and accuracy of generating three views.

[0065] In this embodiment, when generating three views, the user inputs a reference view corresponding to the text prompt, and then obtains the image features of the reference view through the image encoder. In a preferred embodiment, the image encoder is a convolutional neural network, which can process multiple types of reference views, including images of different angles, different lighting conditions, and different resolutions. This generalization ability makes the three-view generation method more flexible and practical, and can cope with various complex practical application scenarios. In addition, the convolutional neural network can quickly extract image features through efficient convolution and pooling operations, making the three-view generation method faster when processing reference views, thereby accelerating the entire three-view generation process.

[0066] In this embodiment, by introducing the reference view provided by the user as input, the specific form, size ratio and detail features of the target three-dimensional image can be captured more directly. The above-mentioned image features, as an important basis for generating three-view images, can significantly improve the accuracy of the generated results, making the generated three-view images closer to the appearance of the actual object. At the same time, image encoders such as convolutional neural networks have powerful image processing capabilities, which can automatically extract key features in images and understand the contextual relationships between these features. This capability enables the system to better understand the information in the reference view and apply it to the process of generating other views, thereby generating more coherent and consistent three-view images.

[0067] In this application, by setting up a three-view generation module and designing relevant algorithms, the three-view generation module can be combined with any generation model for training, so that the finally trained generation model can generate other views with reference to the reference view and text prompt provided by the user, and iterate in a loop to continuously refine the details. Specifically, the three-view generation module includes a self-attention module, a cross-attention module and a feedback module. By training the above three modules in a loop, a three-view generation model is generated.

[0068] S200, inputting the first image feature into a self-attention module to obtain a first attention;

[0069] In some of these embodiments, the self-attention module includes a first normalization layer and a self-attention layer;

[0070] The above step S200 includes:

[0071] Inputting the first image feature into the first normalization layer to obtain a first standard feature;

[0072] Adding the first image feature and the first standard feature to obtain a self-attention layer input feature;

[0073] The input feature of the self-attention layer is input into the self-attention layer to perform self-attention calculation to obtain the first attention.

[0074] In this embodiment, the self-attention module is used to learn the image features of the first image features that are related to the reference view, so that the three views finally generated can fit the reference view, improve the generation of high-quality three views, and thus ensure the correctness of the three views. For example, if the main view of a cylinder is used as a reference view, no triangles will appear in the three views finally generated.

[0075] In this embodiment, the first normalization layer helps the three-view generation model learn a more robust feature representation by normalizing the first image feature input to the self-attention module, thereby improving the generalization ability of the three-view generation model on different data sets. In addition, the first normalization layer can accelerate the training process and alleviate the problem of gradient disappearance or gradient explosion to a certain extent, making the three-view generation model easier to optimize during the training process. At the same time, the self-attention layer reduces the dependence on time or space information in the traditional sequence model by directly calculating the correlation between elements in the sequence, thereby possibly reducing the training time required for the three-view generation model. In a preferred embodiment, the first normalization layer consists of two 3×3 convolutional layers, a Relu activation function and a batch normalization layer.

[0076] In this embodiment, the self-attention module that combines the self-attention mechanism with the normalization layer has high flexibility and scalability in design. The modular design allows users to easily adjust or expand the model according to the needs of specific tasks to adapt to different application scenarios and data sets.

[0077] In this embodiment, the first image feature is directly extracted from the reference view through the image encoder, and contains the original information and details of the reference view, while the first standard feature is obtained by normalizing the first image feature, and has stronger semantic information and generalization ability. Adding the two can achieve feature fusion, so that the self-attention layer can process richer and more comprehensive information. At the same time, the addition of the first image feature and the first standard feature can enhance the key features. By combining the features in the first image feature and the first standard feature, the input features of the self-attention layer can retain the original image information while incorporating more contextual information or high-level semantic information, which is beneficial for the subsequent self-attention module to better capture the dependencies within the image.

[0078] In some embodiments, inputting the self-attention layer input feature into the self-attention layer to perform self-attention calculation to obtain a first attention comprises:

[0079] Learning the detail features of the input features of the self-attention layer through the detail information matrix, and learning the texture features of the input features of the self-attention layer through the texture feature matrix;

[0080] The calculation formula of the first attention is as follows:

[0081]

[0082] Among them, At1 represents the first attention, I1 represents the input feature of the self-attention layer, W represents the texture feature matrix, X represents the detail information matrix, and * represents bitwise multiplication.

[0083] In this embodiment, the self-attention layer is mainly used to learn the detail information and texture features in the reference view. Therefore, a detail information matrix and a texture feature matrix are introduced in the self-attention layer. By learning the detail information and texture features in the two matrices respectively, and learning and integrating the detail information and texture features at the same time, a more comprehensive and in-depth image representation is constructed.

[0084] S300, adding the first attention to the first image feature to obtain a second image feature;

[0085] In this embodiment, the first attention is the attention weight calculated by the self-attention mechanism, which is used to guide the three-view generation model to pay more attention to the detail information and texture features in the reference view, and then by further adding the first attention to the first image feature, the attention weight is combined with the original image feature of the reference view, which can further guide the three-view generation model to learn more effective and useful feature representation, further improve the generation of high-quality three views and ensure the correctness of the generated three views.

[0086] S400, inputting the text feature and the second image feature into a cross-attention module to obtain a second attention;

[0087] In some of these embodiments, the cross-attention module includes a second normalization layer and a cross-attention layer;

[0088] The above step S400 includes:

[0089] Inputting the second image feature into the second normalization layer to obtain a second standard feature;

[0090] Perform feature correlation calculation on the second standard feature and the text feature to obtain a cross-attention layer input feature;

[0091] The cross-attention layer input feature is input into the cross-attention layer to perform cross-attention calculation to obtain a second attention.

[0092] In this embodiment, the cross-attention module is used to measure the similarity between the image features of the reference view and the text features of the text prompt, so as to ensure the correlation between the generated three views and the reference view and the text prompt. For example, a three-view image of a cylinder related to the reference view and the text prompt is finally generated by taking the main view of the cylinder as the reference view and the text prompt.

[0093] In this embodiment, the second image feature input into the cross-attention module is normalized through the second normalization layer, thereby accelerating the training and improving the stability of the three-view generation model. It can be understood that the second normalization layer has the same structure as the first normalization layer, both of which include 2 3×3 convolutional layers, a Relu activation function and a batch normalization layer.

[0094] In some embodiments, the calculation formula for obtaining the cross-attention layer input feature is as follows:

[0095]

[0096] Among them, I2 represents the second image feature, I3 represents the text feature, I4 represents the cross-attention layer input feature, M1 and M2 represent learnable mask matrices, and * represents bitwise multiplication.

[0097] Since the second image features include the detail features and texture features in the reference view, for the input across the attention layer, it is necessary to comprehensively consider the relationship between the detail features and the texture features and the text features, thereby improving the correlation between the reference view and the text prompt, thereby improving the quality and correctness of the generated three views. Therefore, in this application, it is necessary to perform feature correlation calculations on the second standard features and the text features. Specifically, two mask matrices M1 and M2 that can be used for learning are introduced to enable the three-view generation model to more accurately control the interaction between the detail features and the texture features and the text features when transmitting the above-mentioned detail features, texture features and text features across the attention layer. By learning and adjusting the mask matrix, the three-view generation model can enhance the correlation between those features that are beneficial to improving the quality and correctness of the generated three views.

[0098] In some embodiments, inputting the cross-attention layer input feature into the cross-attention layer for cross-attention calculation to obtain the second attention includes:

[0099] The correlation between the text feature and the detail feature of the second standard feature is learned through a detail correlation matrix, and the correlation between the text feature and the texture feature of the second standard feature is learned through a texture correlation matrix;

[0100] The calculation formula of the second attention is as follows:

[0101]

[0102] Among them, At2 represents the second attention, I4 represents the cross-attention layer input feature, Y represents the detail correlation matrix, T represents the texture correlation matrix, M3 represents the learnable mask matrix, and * represents bitwise multiplication.

[0103] In this embodiment, a detail correlation matrix and a texture correlation matrix are introduced across the attention layer to measure the similarity between the image features of the reference view and the text features of the text prompt, further ensuring the relevance and correctness of the three views.

[0104] In this embodiment, on the basis of introducing the detail correlation matrix and the texture correlation matrix, a mask matrix M3 is also introduced, so that the three-view generation model can more flexibly adjust the weights and relationships between the cross-attention layer input features and the detail features and texture features when processing the input cross-attention layer input features, thereby enabling the three-view generation model to better capture the key information of the cross-attention layer input features, and generate higher quality and more accurate output results accordingly, that is, the generated three views will be more in line with the requirements of the reference view and text prompts.

[0105] It can be understood that in a three-view generation method of the present application, all the matrices introduced, namely the detail information matrix, the texture feature matrix, the detail correlation matrix, the texture correlation matrix and the three mask matrices are learnable, avoiding the influence of data initialization on the results, and there is no need to select a suitable control matrix by yourself.

[0106] S500, adding the second attention feature, the text feature and the second image feature to obtain a third image feature;

[0107] In this embodiment, the second attention is the attention weight calculated by the cross-attention mechanism, which is used to guide the three-view generation model to pay more attention to the detail information in the reference view and the correlation between the texture features and the text features in the text prompts. Therefore, the second attention is added to the text features and the second image features, so that the attention weight is combined with the processed second image features and the text features, which can further guide the three-view generation model to learn more effective and useful feature representations, further improve the generation of high-quality three views and ensure the correctness of the generated three views.

[0108] S600: Input the third image feature into a feedback module to obtain a three-view image.

[0109] In some of these embodiments, the feedback module includes a third normalization layer and a feedback layer;

[0110] The above step S600 includes:

[0111] Inputting the third image feature into the third normalization layer to obtain a third standard feature;

[0112] The third standard feature is input into the feedback layer to obtain a three-view image.

[0113] In this embodiment, the feedback module is subdivided into two main parts: a third normalization layer and a feedback layer. The third normalization layer and the feedback layer work together to achieve the conversion from image features to three-view images. Specifically, the main function of the third normalization layer is to standardize the input third image features, which helps to eliminate the scale differences between different data and makes the three-view generation model easier to learn. At the same time, the normalized third standard features are then input to the feedback layer, and the feedback layer converts the above third standard features into the final three-view images. It can be understood that the conversion process of the three-view image involves complex nonlinear transformations, decoding operations or view synthesis techniques, depending on the design and implementation of the feedback layer. The output of the feedback layer is a three-view image, that is, an image of the same object presented from different perspectives (such as the front, side, and top).

[0114] See also Figure 2As shown, the present invention also provides a three-view generating system, the system comprising:

[0115] The first processing module 201 is used to input the text prompt into a text encoder to obtain text features, and input the reference view into an image encoder to obtain first image features;

[0116] The second processing module 202 is used to input the first image feature into a self-attention module to obtain a first attention;

[0117] The third processing module 203 is used to add the first attention and the first image feature to obtain a second image feature;

[0118] The fourth processing module 204 is used for inputting the text feature and the second image feature into a cross-attention module to obtain a second attention;

[0119] A fifth processing module 205: configured to add the second attention, the text feature and the second image feature to obtain a third image feature;

[0120] The sixth processing module 206 is used to input the third image feature into the feedback module to obtain a three-view image.

[0121] It is understandable that if Figure 1 The contents of the three-view image generation method embodiment shown in the figure are applicable to the three-view image generation system embodiment. The functions specifically implemented by the three-view image generation system embodiment are similar to those in the figure. Figure 1 The three-view generating method embodiment shown is the same as that shown in FIG. Figure 1 The beneficial effects achieved by the three-view image generation method embodiment shown are also the same.

[0122] It should be noted that the information interaction, execution process and other contents between the above-mentioned systems are based on the same concept as the embodiment of the method of the present invention. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0123] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the system can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0124] See also Figure 3 As shown, an embodiment of the present invention further provides a computer device 3, comprising: a memory 302 and a processor 301 and a computer program 303 stored in the memory 302. When the computer program 303 is executed on the processor 301, a three-view generating method as described in any one of the above methods is implemented.

[0125] The computer device 3 may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The computer device 3 may include, but is not limited to, a processor 301 and a memory 302. Those skilled in the art will appreciate that Figure 3 It is only an example of computer device 3 and does not constitute a limitation on computer device 3. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components, for example, it may also include input and output devices, network access devices, etc.

[0126] The processor 301 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0127] In some embodiments, the memory 302 may be an internal storage unit of the computer device 3, such as a hard disk or memory of the computer device 3. In other embodiments, the memory 302 may also be an external storage device of the computer device 3, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device 3. Further, the memory 302 may also include both an internal storage unit of the computer device 3 and an external storage device. The memory 302 is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of the computer program, etc. The memory 302 may also be used to temporarily store data that has been output or is to be output.

[0128] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the three-view generating method as described in any one of the above methods is implemented.

[0129] In this embodiment, if the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the camera / computer device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. For example, USB flash drive, mobile hard disk, disk or optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0130] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A three-view generating method, It is characterized in that include: Inputting the text prompt into a text encoder to obtain a text feature, and inputting the reference view into an image encoder to obtain a first image feature; Inputting the first image feature into a self-attention module to obtain a first attention; Adding the first attention to the first image feature to obtain a second image feature; Inputting the text feature and the second image feature into a cross-attention module to obtain a second attention; Adding the second attention, the text feature and the second image feature to obtain a third image feature; The third image feature is input into a feedback module to obtain a three-view image.

2. The method according to claim 1, It is characterized in that The self-attention module includes a first normalization layer and a self-attention layer; The step of inputting the first image feature into a self-attention module to obtain a first attention comprises: Inputting the first image feature into the first normalization layer to obtain a first standard feature; Adding the first image feature and the first standard feature to obtain a self-attention layer input feature; The input feature of the self-attention layer is input into the self-attention layer to perform self-attention calculation to obtain the first attention.

3. The method according to claim 2, It is characterized in that The step of inputting the self-attention layer input feature into the self-attention layer for self-attention calculation to obtain a first attention comprises: Learning the detail features of the input features of the self-attention layer through the detail information matrix, and learning the texture features of the input features of the self-attention layer through the texture feature matrix; The calculation formula of the first attention is as follows: Among them, At1 represents the first attention, I 1 Represents the input features of the self-attention layer, W represents the texture feature matrix, X represents the detail information matrix, and * represents bitwise multiplication.

4. The method according to claim 1, It is characterized in that The cross-attention module includes a second normalization layer and a cross-attention layer; The step of inputting the text feature and the second image feature into a cross-attention module to obtain a second attention comprises: Inputting the second image feature into the second normalization layer to obtain a second standard feature; Perform feature correlation calculation on the second standard feature and the text feature to obtain a cross-attention layer input feature; The cross-attention layer input feature is input into the cross-attention layer to perform cross-attention calculation to obtain a second attention.

5. The method according to claim 4, It is characterized in that The calculation formula for obtaining the cross-attention layer input features is as follows: Among them, I 2 represents the second image feature, I 3 Represents text features, I 4 represents the input features across the attention layer, M 1 and M 2 represents a learnable mask matrix and * represents bitwise multiplication.

6. The method according to claim 4, It is characterized in that The step of inputting the cross-attention layer input feature into the cross-attention layer for cross-attention calculation to obtain a second attention comprises: The correlation between the text feature and the detail feature of the second standard feature is learned through a detail correlation matrix, and the correlation between the text feature and the texture feature of the second standard feature is learned through a texture correlation matrix; The calculation formula of the second attention is as follows: Among them, At2 represents the second attention, I 4 represents the input features across the attention layer, Y represents the detail correlation matrix, T represents the texture correlation matrix, M 3 represents a learnable mask matrix and * represents bitwise multiplication.

7. The method according to claim 1, It is characterized in that The feedback module includes a third normalization layer and a feedback layer; The step of inputting the third image feature into a feedback module to obtain a three-view image comprises: Inputting the third image feature into the third normalization layer to obtain a third standard feature; The third standard feature is input into the feedback layer to obtain a three-view image.

8. A three-view image generation system, It is characterized in that include: A first processing module: used for inputting the text prompt into a text encoder to obtain text features, and inputting the reference view into an image encoder to obtain first image features; A second processing module: used for inputting the first image feature into a self-attention module to obtain a first attention; A third processing module: configured to add the first attention to the first image feature to obtain a second image feature; A fourth processing module: used for inputting the text feature and the second image feature into a cross-attention module to obtain a second attention; A fifth processing module: configured to add the second attention feature, the text feature and the second image feature to obtain a third image feature; The sixth processing module is used to input the third image feature into the feedback module to obtain a three-view image.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program. It is characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images