An image processing method, apparatus, device, and storage medium

Abstract

This application discloses an image processing method, apparatus, device, and storage medium. The method includes: acquiring input data, which includes data of multiple modalities, and the input data is used to generate a three-dimensional scene; generating a mixed modality graph based on the input data, the mixed modality graph including multiple nodes and multiple edges between the nodes, each node representing an object, and each edge representing the relationship between two nodes corresponding to each edge; generating a three-dimensional scene based on the mixed modality graph, the content of the three-dimensional scene including the geometric shape information of the object corresponding to each node and the spatial position information of the object corresponding to each node in the three-dimensional scene. This allows for flexible processing of data of different modalities, improves the adaptability to input data, significantly enhances the support for complex user-inputted data, and enables the generation of three-dimensional scenes with high spatial consistency and geometric detail.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to an image processing method, apparatus, device, and storage medium. Background Technology

[0002] Indoor 3D scene generation technology is an important research direction in virtual reality, interior design, and smart environment construction. The goal of this technology is to generate highly realistic and semantically consistent 3D scenes based on user input data.

[0003] In practical applications, most 3D scene generation technologies only support text input, that is, they generate corresponding 3D scenes based on text data input by users. Therefore, they are inadequate when dealing with complex user input and have limitations. Summary of the Invention

[0004] This application provides an image processing method, apparatus, device, and storage medium to support user input of data in different modalities and to flexibly process data in different modalities, thereby improving adaptability to complex input data.

[0005] In a first aspect, embodiments of this application provide an image processing method, including:

[0006] The input data is acquired, which includes data from multiple modalities, and the input data is used to generate a three-dimensional scene.

[0007] A mixed modality graph is generated based on the input data. The mixed modality graph includes multiple nodes and multiple edges between the multiple nodes. Each of the multiple nodes is used to represent an object, and each of the multiple edges is used to represent the relationship between the two nodes corresponding to each edge.

[0008] The three-dimensional scene is generated based on the mixed modal graph. The content of the three-dimensional scene includes the geometric shape information of the object corresponding to each node and the spatial position information of the object corresponding to each node in the three-dimensional scene.

[0009] Optionally, the plurality of nodes includes a first node, a second node, a third node, and a fourth node. The features of the first node include the object category features and text features of the first node. The features of the fourth node include the object category features, text features, and visual features of the fourth node. There is no edge between the second node and the third node.

[0010] The generation of the mixed modality map based on the input data includes:

[0011] Feature extraction is performed on the input data to obtain a first mixed modality graph, which includes each node, the features of each node, each edge, and the features of each edge.

[0012] The text features of the first node in the first mixed modality graph are enhanced to obtain a second mixed modality graph. The enhancement process is used to convert the text features of the first node into visual features.

[0013] Relationship prediction is performed on the features corresponding to the second node and the third node in the second mixed modality graph to obtain the mixed modality graph. The relationship prediction is used to generate the features of the edges between the second node and the third node to predict the relationship between the second node and the third node.

[0014] Optionally, the input data includes text data and image data, and the step of extracting features from the input data to obtain a first mixed modality map includes:

[0015] The visual language model is used to extract features from the text data and the image data to obtain the multiple nodes. The visual language model is also used to extract features from the text data and image features to obtain each node, the features included in each node, each edge, and the features of each edge.

[0016] The first mixed modal graph is generated based on each node, the features included in each node, each edge, and the features of each edge.

[0017] Optionally, the step of enhancing the text features of the first node in the first mixed modality graph to obtain the second mixed modality graph includes:

[0018] The text features of the first node are encoded by an encoder to obtain a latent vector;

[0019] The latent vectors are quantized using a codebook to obtain quantized latent vectors;

[0020] The quantized latent vector is decoded by a decoder to obtain the visual features of the first node;

[0021] The second mixed modality map is generated based on the visual features of the first node and the first mixed modality map.

[0022] Optionally, the step of predicting the relationship between the features corresponding to the second node and the third node in the second mixed modality graph to obtain the mixed modality graph includes:

[0023] Construct a triplet, which sequentially includes the features of the second node, the relationship between the second node and the third node, and the features of the third node, with the relationship between the second node and the third node padded with zeros;

[0024] The triplet is input into the relation prediction model, and the relation prediction model outputs the features of the edge between the second node and the third node.

[0025] The mixed modality graph is generated based on the features of the edge between the second node and the third node and the second mixed modality graph.

[0026] Optionally, generating the 3D scene based on the mixed modality graph includes:

[0027] Using the hybrid modal graph as a condition for the shape branch in the bi-branch diffusion model, sampling and denoising are performed through the shape branch to obtain the geometric shape information of the object corresponding to each node;

[0028] Using the hybrid modal graph as a condition for the layout branch in the dual-branch diffusion model, sampling and denoising are performed through the layout branch to obtain the spatial position information of the object corresponding to each node in the three-dimensional scene.

[0029] Optionally, the loss function of the shape branch is used to minimize the deviation between the real noise of the object shape in the 3D scene and the predicted noise of the shape branch, and the loss function of the layout branch is used to minimize the deviation between the real noise of the object layout in the 3D scene and the predicted noise of the layout branch. The loss function of the dual-branch diffusion model is determined based on the weights corresponding to the loss function of the shape branch, the loss function of the shape branch, the weights of the loss function of the layout branch, and the loss function of the layout branch.

[0030] Secondly, embodiments of this application provide an image processing apparatus, including:

[0031] An input acquisition module is used to acquire input data, which includes data in multiple modalities, and the input data is used to generate a three-dimensional scene.

[0032] A mixed modality graph generation module is used to generate a mixed modality graph based on the input data. The mixed modality graph includes multiple nodes and multiple edges between the multiple nodes. Each of the multiple nodes is used to represent an object, and each of the multiple edges is used to represent the relationship between the two nodes corresponding to each edge.

[0033] A 3D scene generation module is used to generate the 3D scene based on the mixed modality graph. The content of the 3D scene includes the geometric shape information of the object corresponding to each node and the spatial position information of the object corresponding to each node in the 3D scene.

[0034] Thirdly, embodiments of this application provide an electronic device, the device including: a processor, a memory, and a system bus;

[0035] The processor and the memory are connected via the system bus;

[0036] The memory is used to store a program, the program including instructions that, when executed by the processor, cause the processor to perform any of the implementation steps of the image processing method described above.

[0037] Fourthly, embodiments of this application provide a computer-readable storage medium storing instructions that, when executed on an electronic device, cause the electronic device to perform any of the implementation steps of the above-described image processing method.

[0038] As can be seen from the above technical solutions, the embodiments of this application have the following advantages:

[0039] In this embodiment, firstly, input data can be acquired, including data from multiple modalities, and this input data is used to generate a 3D scene. Then, a mixed modality graph can be generated based on the input data. This mixed modality graph includes multiple nodes and multiple edges between the nodes. Each node represents an object, and each edge represents the relationship between the two nodes corresponding to that edge. Thus, a 3D scene can be generated based on the mixed modality graph. The content of the 3D scene includes the geometric shape information of the object corresponding to each node and the spatial position information of the object corresponding to each node within the 3D scene. Since the mixed modality graph is generated based on data from multiple modalities, this scheme can support user input of data from different modalities, converting the data from different modalities into a mixed modality graph for 3D scene generation. Therefore, it can flexibly handle data from different modalities, improve the adaptability to input data, and significantly enhance the support capability for complex user input data. Furthermore, the image generated based on the mixed modality graph can comprehensively process the spatial layout and geometry of objects under complex input conditions, thereby generating a 3D scene with high spatial consistency and geometric detail. Attached Figure Description

[0040] Figure 1 A flowchart illustrating an image processing method provided in an embodiment of this application;

[0041] Figure 2A flowchart illustrating the generation of a 3D scene using a dual-branch diffusion model is provided in this application embodiment.

[0042] Figure 3 This is a schematic diagram of the structure of an image processing device provided in an embodiment of this application. Detailed Implementation

[0043] As mentioned earlier, in practical applications, most 3D scene generation technologies only support text input, that is, generate corresponding 3D scenes based on user-inputted text data. However, they are difficult to integrate user-inputted image data or mixed modal data (such as a combination of text data and image data). Therefore, they are inadequate and have limitations when dealing with complex user input.

[0044] Based on this, in order to solve the above problems, this application provides an image processing method, which includes: first, acquiring input data, which includes data of multiple modalities, and the input data is used to generate a three-dimensional scene; then, generating a mixed modality graph based on the input data, the mixed modality graph including multiple nodes and multiple edges between the multiple nodes, each of the multiple nodes representing an object, and each of the multiple edges representing the relationship between the two nodes corresponding to each edge, thus generating a three-dimensional scene based on the mixed modality graph, wherein the content of the three-dimensional scene includes the geometric shape information of the object corresponding to each node and the spatial position information of the object corresponding to each node in the three-dimensional scene.

[0045] Since mixed modal graphs are generated based on data from multiple modalities, this scheme can support user input of data from different modalities, converting the data into mixed modal graphs for 3D scene generation. Therefore, it can flexibly handle data from different modalities, improving adaptability to input data and significantly enhancing its ability to support complex user input. Furthermore, images generated based on mixed modal graphs can comprehensively process the spatial layout and geometry of objects under complex input conditions, thereby generating 3D scenes with high spatial consistency and geometric detail.

[0046] It should be noted that the embodiments of this application do not limit the executing entity of the image processing method. For example, the image processing method of the embodiments of this application can be applied to image processing devices such as terminal devices or servers. The terminal device can be an electronic device such as a smartphone, computer, personal digital assistant (PDA), or tablet computer. The server can be a standalone server, a cluster server, or a cloud server.

[0047] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0048] Figure 1 This is a flowchart illustrating an image processing method provided in an embodiment of this application. (In conjunction with...) Figure 1 As shown, the image processing method provided in this application embodiment uses a corresponding image processing device as the execution subject to describe the specific implementation of the scheme. The image processing method may include the following steps S101-S103.

[0049] S101: Acquire input data, which includes data from multiple modalities, and this input data is used to generate a 3D scene.

[0050] In this embodiment, the multi-modal data refers to text data and image data. Accordingly, in a specific implementation, the image processing device can receive multi-modal data input by the user through a terminal device as input data, and use this data to generate a three-dimensional scene.

[0051] S102: Generate a mixed modality graph based on the input data. The mixed modality graph includes multiple nodes and multiple edges between the nodes. Each node represents an object, and each edge represents the relationship between the two nodes corresponding to each edge.

[0052] In this embodiment, the hybrid modal graph is the core structure used to integrate data from multiple modalities. By representing objects and relationships in the input data as corresponding nodes and edges, it provides a unified expression framework for 3D scene generation, thereby enabling the integration of data from multiple modalities. For ease of understanding, the generation process of the hybrid modal graph is described below with reference to a possible implementation method.

[0053] In one possible implementation, the aforementioned multiple nodes may include a first node, a second node, a third node, and a fourth node. The features of the first node include its object category features and text features, while the features of the fourth node include its object category features, text features, and visual features. This means that the first node lacks its visual graph features. Furthermore, there is no edge between the second and third nodes. In other words, there is no corresponding relationship between the second and third nodes.

[0054] Furthermore, among the multiple nodes, a fifth node may also be included, whose features include the visual features of the fifth node. Therefore, based on the features of the nodes, the above multiple nodes can be divided into three types: one type includes only visual features, such as the fifth node; another type includes object category features and text features, such as the first node; and yet another type includes object category features, text features, and visual features, such as the fourth node.

[0055] It should be noted that the embodiments of this application only use the first node, the second node, the third node, the fourth node, and the fifth node as examples to illustrate the features contained in multiple nodes, and do not specifically limit the number of the first node, the second node, the third node, the fourth node, and the fifth node.

[0056] Based on this, the process of generating a mixed modality diagram based on the input data can include the following steps 21-23.

[0057] Step 21: Extract features from the input data to obtain a first mixed modality graph, which includes each node, the features of each node, each edge, and the features of each edge.

[0058] As mentioned earlier, the input data includes text data and image data. Accordingly, in specific implementation, firstly, feature extraction can be performed on the text data and image data using a visual language model, resulting in multiple nodes. Furthermore, feature extraction can be performed on the text data and image features using a visual language model to obtain the features included in each node, each edge, and the features of each edge.

[0059] Multiple nodes can form a node set V. m Multiple edges can form an edge set E. m V m ={v1,v2,…,v N}, where N is the number of nodes, E m ={e i,j |i,j∈{1,…,N},i≠j},e i,j This represents the edge between node i and node j. The characteristics of each edge include the similarity relationship and spatial position relationship between the objects corresponding to each node.

[0060] Taking nodes i and j as examples, when node i is the first node, the features of node i can include object category features. and text features The features of node j can include object category features. and text features and visual features That is, the feature representation of node i is The feature representation of node j is as follows The edge between node i and node j is represented as Where, r i,j This represents the relationship between node i and node j, which can include semantic similarity and spatial location relationships. The similarity relationship describes the similarity between the objects corresponding to node i and node j, such as whether they are similar in style or material. The spatial location relationship describes the spatial position between the objects corresponding to node i and node j, such as their relative distance and relative orientation.

[0061] It should be noted that the methods for obtaining the three features of the above-mentioned nodes are not specifically limited in this embodiment. For example, category features can be extracted through an embedding layer, text features can be extracted through a text encoder, such as CLIP or LLaMa, and visual features can be extracted through an image feature encoder, such as CLIP or DINO.

[0062] Additionally, it should be noted that if a node lacks object category features, text features, or visual features, zero padding can be used to ensure consistency of feature categories.

[0063] Next, a first mixed modal graph can be generated based on each node, the features included in each node, each edge, and the features of each edge. This first mixed modal graph G... m It can be formally defined as G m =(V m E m ).

[0064] In this way, generating a first mixed modality graph based on multi-modal input data can significantly improve the flexibility and scalability of the data. It can support text, images, and their combinations, adapting to diverse user needs, and also provides high-quality input features for subsequent augmentation and relationship prediction processing.

[0065] Step 22: Enhance the text features of the first node in the first mixed modality graph to obtain the second mixed modality graph. The enhancement process is used to convert the text features of the first node into visual features.

[0066] As mentioned earlier, the first node lacks visual features. Therefore, in this embodiment, enhancement processing can be performed on nodes lacking visual features to provide high-quality visual features, thereby addressing the shortcomings of existing 3D scene generation schemes in terms of geometric control capabilities.

[0067] As an example, taking node i as the first node, the features of the first node can be enhanced by a visual enhancement module. This visual enhancement module can adopt a Q-VAE architecture (a variant of a variational autoencoder), in which an encoder, decoder, and codebook are included.

[0068] Accordingly, firstly, the text features of the first node can be encoded by an encoder to obtain the latent vector, as shown in formula (1) below:

[0069]

[0070] in, Let E be the latent vector and E be the encoder. The text features of the first node.

[0071] Next, the latent vectors obtained above can be quantized using a codebook to obtain quantized latent vectors. The codebook can be represented as follows: This means that the codebook C includes K embedding vectors e. k Accordingly, the quantization process via the codebook can be represented as determining the corresponding values ​​from these K embedding vectors. The n closest embedding vectors are shown in the following formula (2):

[0072]

[0073] in, For the quantized latent vector, For the j-th embedding vector, For potential vectors, Let be the l-th embedding vector.

[0074] Based on the above formula (2), in the quantization process, the latent vector can be calculated first. The squared Euclidean distance between the vectors and each of the K embedding vectors is calculated. Then, based on the squared distance of each embedding vector, n embedding vectors are selected from the K embedding vectors. The sum of the squared distances of these n selected embedding vectors is minimized, and these n embedding vectors constitute the quantized latent vector.

[0075] Then, the quantized latent vector can be decoded by the decoder to obtain the visual features of the first node, as shown in the following formula (3):

[0076]

[0077] in, Let D be the visual feature of the first node, and D be the decoder. This is the quantized latent vector.

[0078] In this way, based on the visual features of the first node and the first mixed modality graph G, m Generate the second mixed mode diagram That is, by further generating a visually enhanced second mixed modality map from the initial first mixed modality map, a powerful support is provided for generating high-quality, geometrically accurate 3D objects.

[0079] Furthermore, in this embodiment, the training objective of the aforementioned visual enhancement module is to maximize the Evidence Lower Bound (ELBO) of the data to optimize the generation quality of visual features. Accordingly, the objective function of this visual enhancement module can be expressed as follows: (4)

[0080]

[0081] in, Let q be the value of the objective function. E (h|t) is the latent vector distribution, p D (u|h) is the likelihood probability generated from visual features, D KL The Kullback-Leibler divergence measures the difference between the latent distribution and the Gaussian prior distribution p(h), and β is an important parameter used to weigh the divergence term. Furthermore, to address the non-differentiability issue of the quantization process, the Gumbel-Softmax relaxation technique is employed to optimize the ELBO objective function.

[0082] Step 23: Perform relation prediction on the features corresponding to the second and third nodes in the second mixed modality graph to obtain the mixed modality graph. The relation prediction is used to generate the features of the edges between the second and third nodes to predict the relationship between the second and third nodes.

[0083] As mentioned earlier, there is a lack of a corresponding relationship between the second and third nodes. Therefore, in this embodiment, a more reasonable scene layout can be generated by predicting the relationship between each pair of nodes that lack a relationship. This addresses the problem of inaccurate scene layout generation results due to the lack of relational information in existing 3D scene generation schemes.

[0084] As an example, a relationship prediction module can be used to predict the relationship between the features of the second node and the features of the third node. Here, the structure of the relationship prediction module in this embodiment is not specifically limited, and it can be constructed using any existing or future network capable of extracting triplet features.

[0085] Accordingly, in this embodiment, the second node is taken as node m, and the third node is taken as node n. First, a triplet can be constructed, which can be used as input to the relation prediction module. The triplet can sequentially include the features of the second node, the relationship between the second and third nodes, and the features of the third node, i.e. Since the relationship between the second and third nodes is missing, it is filled with zeros. In other words, this triplet can be represented as... This ensures the consistency of the feature space.

[0086] Next, the triples can be input into the relation prediction model, which then outputs the features of the edge between the second and third nodes. Here, the features of the edge between the second and third nodes output by the relation prediction module represent the relationship between them.

[0087] In this way, we can use the features of the edge between the second node and the third node and the second mixed modality graph. Generate mixed mode diagram That is, a second mixed modal graph is used to further generate a mixed modal graph that completes the relationships, thereby capturing the relationships between nodes and effectively improving the overall consistency and rationality of the scene layout.

[0088] Furthermore, in this embodiment, the aforementioned relationship prediction module can be trained using a cross-entropy loss function, which can be expressed as shown in the following formula (5):

[0089]

[0090] in, Here, is the cross-entropy loss value, N is the number of node pairs, C is the number of relation classes, and y is... ic One-hot encoding of real relationships, This is the probability of the relationship category predicted by the relationship prediction module. By minimizing this loss function, the relationship prediction module can accurately predict the relationships between nodes, thus efficiently filling in missing relationships.

[0091] S103: Generate a 3D scene based on a mixed modal graph. The content of the 3D scene includes the geometric shape information of the object corresponding to each node and the spatial position information of the object corresponding to each node in the 3D scene.

[0092] In this embodiment, a two-branch diffusion model can be used to generate a 3D scene. Its main objective is to generate a 3D scene that meets the conditions of a mixed modality graph, which may include the spatial layout and geometry of objects. Accordingly, combined with... Figure 2As shown, the dual-branch diffusion model can include two branches: a layout branch and a shape branch. The layout branch is responsible for generating the spatial position information of the object, such as the center position, size, and rotation angle of the object, while the shape branch is responsible for generating the three-dimensional geometry of the object.

[0093] Furthermore, in order to effectively realize information exchange and relationship modeling between nodes, this bi-branch diffusion model can be implemented using a graph encoder E. g As shown in the figure, convolutional neural networks (GCNs) update the latent representation of mixed modality graphs. For ease of understanding, the process of generating a 3D scene based on a mixed modality graph is illustrated below with an example of a possible implementation.

[0094] As one possible implementation, a hybrid modal graph can be used as a condition for the shape branching process. Sampling and denoising are then performed through the shape branching to obtain the geometric shape information of the object corresponding to each node. Furthermore, using the hybrid modal graph as a condition for the layout branching process, sampling and denoising are then performed through the layout branching to obtain the spatial position information of the object corresponding to each node in the 3D scene.

[0095] In practical implementation, firstly, the mixture modality graph can be used as conditional input for the shape and layout branches. This allows the shape branch to know the initial shape of the object to be generated, and the layout branch to know the initial layout. Next, sampling data can be obtained from a Gaussian distribution or other suitable noise distribution. This sampling data reflects the initial geometric shape and initial spatial position information of the object. Then, using this sampling data as input, progressive denoising is performed through the shape and layout branches, transforming the sampled data into the final shape and layout—that is, the geometric shape information and spatial position information of the object corresponding to each node in the 3D scene.

[0096] Furthermore, during the model training phase, firstly, real data can be acquired, namely, the geometric shape and spatial location information of objects in a real 3D scene. Next, noise can be added to the real data. In this way, the bi-branch diffusion model can use a mixture modality diagram as a condition to predict the noise added to the real data, and use a loss function to measure the difference between the accuracy of the model's prediction and the actual noise. Thus, by continuously optimizing the model parameters to minimize the loss function, a bi-branch diffusion model with accurate noise prediction can be obtained.

[0097] More specifically, the shape branch uses a truncated signed distance field (TSDF) to represent the object shape, and encodes it using a pre-trained and frozen VQ-VAE model to generate an initial latent representation. The final shape is then gradually generated through a denoising process.

[0098] During the training phase of this shape branch, for each denoising time step t∈{1,2,…,T}, the graph encoder E g It is possible to represent the latent representation and potential map Processing generates an updated latent representation and potential map Among them, potential graph Based on the above mixed mode diagram The resulting updated potential graph The nodes in the array are represented as Next, the updated potential graph described above. Nodes in the ∈ can be used as denoisers. θ The conditional input of the denoiser ∈ θ It can be implemented using 3D UNet (a deep learning architecture for image segmentation) or DiT (a novel model that combines the Transformer architecture and the diffusion model).

[0099] In practical applications, the training objective of this shape branch is to minimize the deviation between the real noise ∈ and the model prediction noise. That is, the loss function of the shape branch is used to minimize the deviation between the real noise of the object shape in the 3D scene and the prediction noise of the shape branch. Accordingly, the loss function of the shape branch can be expressed as follows (6):

[0100]

[0101] in, The loss value for shape branching. Let be the updated latent representation, t be the denoised time step, ∈ be the real noise, and ∈ be the real noise. θ For noise reduction, For the updated potential graph The nodes in the diagram are represented.

[0102] The layout branches are based on the bounding box representation of the object, and each bounding box... Including location size and rotation angle

[0103] During the training phase of this layout branch, to improve the scalability and numerical stability of the training process, the rotation angle is... To improve numerical stability, the rotation angle can be represented using trigonometric functions; that is, the rotation angle can be expressed as... Furthermore, it is possible to... and Normalization is performed.

[0104] In a graph structure, the layout branches are transmitted through the graph encoder E. g By performing relational modeling, an updated potential layout representation can be generated. and node embedding Next, the updated node embeddings and potential layout representations described above can be used as a denoiser ψ. θ The conditional input is processed by the denoiser ψ. θ The denoiser ψ gradually generates the final spatial layout. θ It can be implemented using one-dimensional UNet or Transformer (a deep learning model based on self-attention mechanism).

[0105] In practical applications, the training objective of this layout branch is to minimize the deviation between the real noise ψ and the model's predicted noise. That is, the loss function of the layout branch is used to minimize the deviation between the real noise of the object layout in the 3D scene and the predicted noise of the layout branch. Accordingly, the loss function of the layout branch can be expressed as follows (7):

[0106]

[0107] in, The loss value for the layout branch. Here is the updated potential layout representation, t is the denoised time step, and ψ is the true noise. θ For noise reduction, For the updated potential graph Node embedding in.

[0108] Based on this, in the embodiments of this application, the overall optimization objective of the above-mentioned dual-branch diffusion model can be reflected by the weighted combination of the losses of the shape branch and the layout branch. That is, the loss function of the dual-branch diffusion model is determined based on the weights of the loss function of the shape branch, the loss function of the shape branch, the weights of the loss function of the layout branch, and the loss function of the layout branch. Specifically, it can be shown in the following formula (8):

[0109]

[0110] in, Let α be the loss value of the two-branch diffusion model, and α1 be the weight of the loss function for the placement branch. α1 represents the loss value for the layout branch, and α2 represents the weight of the loss function for the shape branch. This represents the loss value for the shape branch.

[0111] As can be seen from the above steps S101-S103, in this embodiment, firstly, input data can be acquired, which includes data from multiple modalities. This input data is used to generate a 3D scene. Then, a mixed modality graph can be generated based on the input data. This mixed modality graph includes multiple nodes and multiple edges between the nodes. Each node represents an object, and each edge represents the relationship between the two nodes corresponding to each edge. Thus, a 3D scene can be generated based on the mixed modality graph. The content of the 3D scene includes the geometric shape information of the object corresponding to each node and the spatial position information of the object corresponding to each node in the 3D scene. Since the mixed modality graph is generated based on data from multiple modalities, this means that the scheme can support users inputting data from different modalities and then converting the data from different modalities into a mixed modality graph to generate a 3D scene. Therefore, it can flexibly process data from different modalities, improve the adaptability to input data, and thus significantly improve the support capability for complex data input by users. Furthermore, images generated based on mixed modal graphs can comprehensively process the spatial layout and geometry of objects under complex input conditions, thereby generating 3D scenes with high spatial consistency and geometric details.

[0112] Furthermore, based on the image processing method provided in the above embodiments, this application embodiment can also provide an image processing apparatus. The image processing apparatus will now be described in conjunction with the embodiments and accompanying drawings.

[0113] Figure 3 This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of this application. (In conjunction with...) Figure 3 As shown, the image processing apparatus 300 provided in this application embodiment may include:

[0114] The input acquisition module 301 is used to acquire input data, which includes data of multiple modalities, and the input data is used to generate a three-dimensional scene;

[0115] The mixed modality graph generation module 302 is used to generate a mixed modality graph based on the input data. The mixed modality graph includes multiple nodes and multiple edges between the multiple nodes. Each of the multiple nodes is used to represent an object, and each of the multiple edges is used to represent the relationship between the two nodes corresponding to each edge.

[0116] The 3D scene generation module 303 is used to generate the 3D scene based on the mixed modality graph. The content of the 3D scene includes the geometric shape information of the object corresponding to each node and the spatial position information of the object corresponding to each node in the 3D scene.

[0117] Optionally, the plurality of nodes includes a first node, a second node, a third node, and a fourth node. The features of the first node include the object category features and text features of the first node. The features of the fourth node include the object category features, text features, and visual features of the fourth node. There is no edge between the second node and the third node.

[0118] The mixed modality graph generation module 302 includes:

[0119] The feature extraction module is used to extract features from the input data to obtain a first mixed modality graph, wherein the first mixed modality graph includes each node, the features of each node, each edge, and the features of each edge;

[0120] An enhancement processing module is used to enhance the text features of the first node in the first mixed modality graph to obtain a second mixed modality graph. The enhancement processing is used to convert the text features of the first node into visual features.

[0121] The relationship prediction module is used to perform relationship prediction on the features corresponding to the second node and the third node in the second mixed modality graph to obtain the mixed modality graph. The relationship prediction is used to generate the features of the edge between the second node and the third node to predict the relationship between the second node and the third node.

[0122] Optionally, the input data includes text data and image data, and the feature extraction module is specifically used for:

[0123] The visual language model is used to extract features from the text data and the image data to obtain the multiple nodes. The visual language model is also used to extract features from the text data and image features to obtain each node, the features included in each node, each edge, and the features of each edge.

[0124] The first mixed modal graph is generated based on each node, the features included in each node, each edge, and the features of each edge.

[0125] Optionally, the enhanced processing module is specifically used for:

[0126] The text features of the first node are encoded by an encoder to obtain a latent vector;

[0127] The latent vectors are quantized using a codebook to obtain quantized latent vectors;

[0128] The quantized latent vector is decoded by a decoder to obtain the visual features of the first node;

[0129] The second mixed modality map is generated based on the visual features of the first node and the first mixed modality map.

[0130] Optionally, the relationship prediction module is specifically used for:

[0131] Construct a triplet, which sequentially includes the features of the second node, the relationship between the second node and the third node, and the features of the third node, with the relationship between the second node and the third node padded with zeros;

[0132] The triplet is input into the relation prediction model, and the relation prediction model outputs the features of the edge between the second node and the third node.

[0133] The mixed modality graph is generated based on the features of the edge between the second node and the third node and the second mixed modality graph.

[0134] Optionally, the 3D scene generation module 303 is specifically used for:

[0135] Using the hybrid modal graph as a condition for the shape branch in the bi-branch diffusion model, sampling and denoising are performed through the shape branch to obtain the geometric shape information of the object corresponding to each node;

[0136] Using the hybrid modal graph as a condition for the layout branch in the dual-branch diffusion model, sampling and denoising are performed through the layout branch to obtain the spatial position information of the object corresponding to each node in the three-dimensional scene.

[0137] Optionally, the loss function of the shape branch is used to minimize the deviation between the real noise of the object shape in the 3D scene and the predicted noise of the shape branch, and the loss function of the layout branch is used to minimize the deviation between the real noise of the object layout in the 3D scene and the predicted noise of the layout branch. The loss function of the dual-branch diffusion model is determined based on the weights corresponding to the loss function of the shape branch, the loss function of the shape branch, the weights of the loss function of the layout branch, and the loss function of the layout branch.

[0138] Furthermore, embodiments of this application also provide an electronic device, including: a processor, a memory, and a system bus;

[0139] The processor and the memory are connected via the system bus;

[0140] The memory is used to store one or more programs, the one or more programs including instructions that, when executed by the processor, cause the processor to perform any of the implementation steps of the image processing method described above.

[0141] Furthermore, embodiments of this application also provide a computer-readable storage medium storing instructions that, when executed on an electronic device, cause any of the above-described image processing method steps to be implemented.

[0142] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that all or part of the steps in the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network communication device such as a media gateway, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application. It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on describing the differences from other embodiments. The same or similar parts between the various embodiments can be referred to mutually.

[0143] The system disclosed in the embodiments is described simply because it corresponds to the method disclosed in the embodiments; relevant details can be found in the method section.

[0144] It should also be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0145] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image processing method, characterized in that, include: The input data is acquired, which includes data from multiple modalities, and the input data is used to generate a three-dimensional scene. A mixed modality graph is generated based on the input data. The mixed modality graph includes multiple nodes and multiple edges between the multiple nodes. Each of the multiple nodes is used to represent an object, and each of the multiple edges is used to represent the relationship between the two nodes corresponding to each edge. The three-dimensional scene is generated based on the hybrid modal graph. The content of the three-dimensional scene includes the geometric shape information of the object corresponding to each node and the spatial position information of the object corresponding to each node in the three-dimensional scene. The plurality of nodes include a first node, a second node, a third node, and a fourth node. The features of the first node include the object category features and text features of the first node. The features of the fourth node include the object category features, text features, and visual features of the fourth node. There is no edge between the second node and the third node. The generation of the mixed modality map based on the input data includes: Feature extraction is performed on the input data to obtain a first mixed modality graph, which includes each node, the features of each node, each edge, and the features of each edge. The text features of the first node in the first mixed modality graph are enhanced to obtain a second mixed modality graph. The enhancement process is used to convert the text features of the first node into visual features. Relationship prediction is performed on the features corresponding to the second node and the third node in the second mixed modality graph to obtain the mixed modality graph. The relationship prediction is used to generate the features of the edge between the second node and the third node to predict the relationship between the second node and the third node. The generation of the 3D scene based on the mixed modality graph includes: Using the hybrid modal graph as a condition for the shape branch in the bi-branch diffusion model, sampling and denoising are performed through the shape branch to obtain the geometric shape information of the object corresponding to each node; Using the hybrid modal graph as a condition for the layout branch in the dual-branch diffusion model, sampling and denoising are performed through the layout branch to obtain the spatial position information of the object corresponding to each node in the three-dimensional scene.

2. The image processing method according to claim 1, characterized in that, The input data includes text data and image data. The step of extracting features from the input data to obtain a first mixed modality map includes: The visual language model is used to extract features from the text data and the image data to obtain the multiple nodes. The visual language model is also used to extract features from the text data and image features to obtain each node, the features included in each node, each edge, and the features of each edge. The first mixed modal graph is generated based on each node, the features included in each node, each edge, and the features of each edge.

3. The image processing method according to claim 1, characterized in that, The step of enhancing the text features of the first node in the first mixed modality graph to obtain the second mixed modality graph includes: The text features of the first node are encoded by an encoder to obtain a latent vector; The latent vectors are quantized using a codebook to obtain quantized latent vectors; The quantized latent vector is decoded by a decoder to obtain the visual features of the first node; The second mixed modality map is generated based on the visual features of the first node and the first mixed modality map.

4. The image processing method according to claim 1, characterized in that, The step of predicting the relationship between the features corresponding to the second node and the third node in the second mixed modality graph to obtain the mixed modality graph includes: Construct a triplet, which sequentially includes the features of the second node, the relationship between the second node and the third node, and the features of the third node, with the relationship between the second node and the third node padded with zeros; The triplet is input into the relation prediction model, and the relation prediction model outputs the features of the edge between the second node and the third node. The mixed modality graph is generated based on the features of the edge between the second node and the third node and the second mixed modality graph.

5. The image processing method according to claim 1, characterized in that, The loss function of the shape branch is used to minimize the deviation between the real noise of the object shape in the 3D scene and the predicted noise of the shape branch. The loss function of the layout branch is used to minimize the deviation between the real noise of the object layout in the 3D scene and the predicted noise of the layout branch. The loss function of the dual-branch diffusion model is determined based on the weights corresponding to the loss function of the shape branch, the loss function of the shape branch, the weights of the loss function of the layout branch, and the loss function of the layout branch.

6. An image processing apparatus, characterized in that, include: An input acquisition module is used to acquire input data, which includes data in multiple modalities, and the input data is used to generate a three-dimensional scene. A mixed modality graph generation module is used to generate a mixed modality graph based on the input data. The mixed modality graph includes multiple nodes and multiple edges between the multiple nodes. Each of the multiple nodes is used to represent an object, and each of the multiple edges is used to represent the relationship between the two nodes corresponding to each edge. A 3D scene generation module is used to generate the 3D scene based on the mixed modality graph. The content of the 3D scene includes the geometric shape information of the object corresponding to each node and the spatial position information of the object corresponding to each node in the 3D scene. The plurality of nodes includes a first node, a second node, a third node, and a fourth node. The features of the first node include the object category features and text features of the first node. The features of the fourth node include the object category features, text features, and visual features of the fourth node. There is no edge between the second node and the third node. The mixed modality graph generation module includes: The feature extraction module is used to extract features from the input data to obtain a first mixed modality graph, wherein the first mixed modality graph includes each node, the features of each node, each edge, and the features of each edge; An enhancement processing module is used to enhance the text features of the first node in the first mixed modality graph to obtain a second mixed modality graph. The enhancement processing is used to convert the text features of the first node into visual features. The relationship prediction module is used to perform relationship prediction on the features corresponding to the second node and the third node in the second mixed modality graph to obtain the mixed modality graph. The relationship prediction is used to generate the features of the edge between the second node and the third node to predict the relationship between the second node and the third node. The 3D scene generation module is specifically used for: Using the hybrid modal graph as a condition for the shape branch in the bi-branch diffusion model, sampling and denoising are performed through the shape branch to obtain the geometric shape information of the object corresponding to each node; Using the hybrid modal graph as a condition for the layout branch in the dual-branch diffusion model, sampling and denoising are performed through the layout branch to obtain the spatial position information of the object corresponding to each node in the three-dimensional scene.

7. An electronic device, characterized in that, The device includes: a processor, a memory, and a system bus; The processor and the memory are connected via the system bus; The memory is used to store a program, the program including instructions that, when executed by the processor, cause the processor to perform the steps of the image processing method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program, which, when executed by a terminal device, implements the steps of the image processing method according to any one of claims 1 to 5.

Images