Decoupled representation learning method and device based on graph structure and multi-modal large model

Abstract

The application relates to a decoupling representation learning method and equipment based on a graph structure and a multi-modal large model, and the method comprises the following steps: acquiring original image information and performing key point detection; a preset semantic encoder is used to map the key point detection result to a preset hidden space to obtain attribute factors after decoupling of fine-grained representation, wherein each hidden code dimension in the hidden space independently represents an attribute or representation; the attribute factors are initialized, the correlation between the attribute factors is obtained by interacting with a preset large language model based on the key point detection result and the initialized attribute factors; the attribute factors and the correlation between the attribute factors are subjected to structured coding, a preset updater is used to update a pre-established graph structure, and self-supervised optimization is performed based on an image generated by a decoder and the original image information, so that decoupling representation learning is realized. The application realizes semantic perception and decoupling representation in complex data and real scenes.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a decoupled representation learning method and device based on graph structures and multimodal large models. Background Technology

[0002] In recent years, the interpretability of deep learning technology has attracted widespread attention. Whether data-driven artificial intelligence models can truly understand and perceive physical concepts and laws has become crucial.

[0003] Decoupled representation learning aims to decompose representation factors with explicit physical meaning from real data in a way that is understandable to humans, and to control them through corresponding implicit codes. For example, a facial image can be decoupled into factors such as facial features, skin color, and hairstyle; an architectural image can be decoupled into factors such as doors and windows, eaves, and texture; and a medical image can be decoupled into factors such as organs, lesions, and shadows.

[0004] Decoupled representation learning has wide applications in downstream tasks of artificial intelligence. It can provide robust representations of complex variations in data and utilize extracted semantic information for tasks such as recognition, classification, and domain adaptation. Traditional decoupled representation learning models typically use autoencoders or variational autoencoder frameworks to improve data interpretability and the independence of latent factors by minimizing reconstruction errors and regularizing the distribution of the latent space. However, existing techniques such as beta-VAE, FactorVAE, and DEAR still suffer from the following three main problems:

[0005] (1) Traditional decoupled representation learning methods assume that the semantic attributes behind the data are independent of each other and do not affect each other. This assumption is too idealistic, making it difficult for these methods to effectively handle complex datasets.

[0006] (2) Existing decoupled representation methods rely on generated data and data labels, and are usually only applicable to simple datasets. This dependency reduces the model's generalization ability and robustness, making it perform poorly in a wider range of application scenarios.

[0007] (3) Existing methods often ignore the inherent physical relationship between semantic attributes, which limits the application of the model in real-world scenarios and makes it difficult to generate practical value.

[0008] In summary, there is currently a lack of a decoupled representation learning method to solve or partially solve the aforementioned problems. Summary of the Invention

[0009] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a decoupled representation learning method and device based on graph structure and multimodal large model. By fully decoupling the representation factors behind the data, and simultaneously perceiving the inherent physical relationship between these factors, semantic perception and representation decoupling can be achieved in complex data and real-world scenarios.

[0010] The objective of this invention can be achieved through the following technical solutions:

[0011] One aspect of the present invention provides a decoupled representation learning method based on graph structure and multimodal large model, comprising the following steps:

[0012] Step S1: Obtain the original image information and perform key point detection;

[0013] Step S2: Using a preset semantic encoder, the key point detection results are mapped to a preset latent space to obtain the attribute factors after fine-grained representation decoupling, wherein each latent code dimension in the latent space independently represents an attribute or representation.

[0014] Step S3: Initialize the attribute factors. Based on the key point detection results and the initialized attribute factors, obtain the correlation between the attribute factors by interacting with the preset large language model.

[0015] Step S4 involves performing structured encoding based on attribute factors and the relationships between them, updating the pre-established graph structure using a preset updater, and performing self-supervised optimization based on the image generated by the decoder and the original image information to achieve decoupled representation learning.

[0016] As a preferred technical solution, the original image information includes a human face image, and the attribute factors include facial features, hair, and skin color.

[0017] As a preferred technical solution, the graph structure includes nodes and edges, wherein nodes represent fine-grained attribute factors after decoupling, and edges represent the association relationships between attribute factors.

[0018] As a preferred technical solution, in step S4, a preset updater is used to update the graph structure based on preset prompts.

[0019] As a preferred technical solution, the graph structure is a bidirectional weighted graph structure.

[0020] As a preferred technical solution, the updater includes a graph neural network.

[0021] As a preferred technical solution, the semantic encoder is built based on beta-VAE.

[0022] Another aspect of the present invention provides a decoupled representation learning device based on graph structure and multimodal large model, comprising:

[0023] The attribute factor decoupling module is used to map the key point detection results of the original image information to a preset latent space using a preset semantic encoder, thereby obtaining attribute factors after fine-grained representation decoupling. Each latent code dimension in the latent space independently represents an attribute or representation.

[0024] The attribute association perception module is used to initialize the attribute factors, and based on the key point detection results and the initialized attribute factors, obtain the association relationship between the attribute factors by interacting with the preset large language model.

[0025] The graph structure encoding module is used to perform structured encoding based on attribute factors and the relationships between attribute factors, update the pre-established graph structure using a preset updater, and perform self-supervised optimization based on the image generated by the decoder and the original image information to achieve decoupled representation learning.

[0026] In another aspect, an electronic device is provided, comprising: one or more processors and a memory, wherein the memory stores one or more programs, the one or more programs including instructions for executing the aforementioned decoupled representation learning method based on graph structure and multimodal large model.

[0027] In another aspect, the present invention provides a computer-readable storage medium including one or more programs executable by one or more processors of an electronic device, the one or more programs including instructions for performing the aforementioned decoupled representation learning method based on graph structure and multimodal large model.

[0028] Compared with the prior art, the present invention has at least one of the following beneficial effects:

[0029] (1) Achieving semantic perception and representation decoupling in complex data and real-world scenarios: This invention first uses a semantic encoder to map the key point detection results to a preset latent space to obtain attribute factors after fine-grained representation decoupling. Then, it obtains the correlation between attribute factors by interacting with a large language model. Finally, it uses a preset updater to update the pre-established graph structure and perform self-supervised optimization to realize the perception and modeling of the relationship between attributes, thus making it applicable to complex data and real-world scenarios.

[0030] (2) Improve generalization and robustness: This invention uses the semantic knowledge of large models to predict the intrinsic relationship between representation attributes, and continuously updates and optimizes the network parameters through a graph neural network updater, thereby improving the generalization and robustness of the model.

[0031] (3) Enhanced controllability: This invention encodes attributes and their inherent relationships in a bidirectional weighted graph, and represents attributes and inherent relationships through nodes and edges respectively. This encoding method enhances the controllability of the model and is beneficial to the practical application of the algorithm. Attached Figure Description

[0032] Figure 1 This is a schematic diagram illustrating the decoupled representation learning based on graph structure and multimodal large model in the embodiment;

[0033] Figure 2 The method provided in this embodiment yields results on face data and comparisons with existing work.

[0034] Figure 3 The results of the implementation method are verified in other real-world scenarios;

[0035] Figure 4 The examples are verification experiments for multiple multimodal large models.

[0036] Figure 5 for Figure 1 The diagram shows the project details. Detailed Implementation

[0037] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0038] Example 1

[0039] To address the problems of the aforementioned existing technologies, a decoupled representation learning method and device based on graph structures and multimodal large models are provided. This method decouples the representation factors behind the data while simultaneously perceiving the inherent physical relationships between these factors. The model can achieve semantic awareness and representation decoupling in complex data and real-world scenarios, thereby improving its robustness and practicality.

[0040] See Figure 1 This embodiment combines explicit graph structures and multimodal large models to decouple and perceive the representation factors and relationships of natural images. Specifically, it consists of three main modules: an attribute factor decoupling module, an attribute association perception module, and a graph structure encoding module. The design and technical principles of each module are described in detail below:

[0041] Attribute Factor Decoupling Module: After preprocessing by the keypoint detection module, the image is input into a semantic encoder. This semantic encoder is built on beta-VAE, and its goal is to encode high-dimensional image data into a low-dimensional abstract space, namely the latent space. Through training and optimization, each latent code dimension in the latent space can independently represent a specific attribute or representation of the data. This independence means that changing the value of one dimension should not significantly affect the representation of other dimensions, thus achieving fine-grained representation decoupling. The decoupled representation attributes are used as initialization attributes and input into the attribute association perception module along with the initial image for attribute association extraction. Here, attribute initialization refers to the model autonomously predicting the attribute with the largest influence weight for the image and using it as the initialization attribute. Taking a face image as an example, the attributes with the largest learning weight may include facial features, hair, and skin color.

[0042] Attribute Association Awareness Module: The preprocessed image and initialized attribute factors are input into the attribute association awareness module. This module interacts with the large model through prompt engineering; see [link to module]. Figure 5 Using example prompts, leverage their generative capabilities to predict and inductively deduce the relationships between attribute factors. For example, in facial images, beard color might be negatively correlated with age, while gender might be positively correlated with hair length. See also Figure 4 To compare the prediction results of multiple large models for facial attributes, after several sets of comparative experiments, the GPT-4o model was finally selected for association prediction. The attributes obtained through the attribute factor decoupling module and the intrinsic associations obtained through the attribute association perception module will be input into the normalization module for standardization processing to facilitate subsequent graph structure encoding.

[0043] Graph Structure Encoding Module: The graph structure receives input from the two modules mentioned above and performs structured encoding. This graph structure is defined as G = (A, S), where A represents the decoupled attributes, serving as the set of nodes in the graph structure; S represents the inner-relations predicted by the large model, serving as the set of edges in the graph structure. Since the influence between the two attributes is mutual and their influence weights are different, G is defined as a bidirectional weighted graph. During training, the association prediction results of the large model are continuously updated with the edges and weights in G through an updater based on a Graph Neural Network (GNN). Furthermore, the representation output by the graph structure is used to generate images through a decoder, and the entire model is optimized through self-supervised comparison with the original input image. Specifically, the optimization targets include the semantic encoder, decoder, and updater.

[0044] Based on the above modules, the decoupled representation learning method based on graph structure and multimodal large model in this embodiment includes the following steps:

[0045] Step S1: Obtain the original image information and perform key point detection;

[0046] Step S2: Using a semantic encoder, the key point detection results are mapped to a preset latent space to obtain the attribute factors after fine-grained representation decoupling. In the latent space, each latent code dimension independently represents an attribute or representation.

[0047] Step S3: Initialize the attribute factors. Based on the key point detection results and the initialized attribute factors, obtain the correlation between attribute factors by interacting with the preset large language model.

[0048] Step S4 involves performing structured encoding based on attribute factors and the relationships between them, updating the pre-established graph structure using a preset updater, and performing self-supervised optimization based on the image generated by the decoder and the original image information to achieve decoupled representation learning.

[0049] To verify the effectiveness of the solution provided in this embodiment, this solution is compared with FactorVAE and DEAR models. See [link to relevant documentation]. Figure 2 , Figure 2 In the diagram, (a) shows the result corresponding to the FactorVAE model, (b) shows the result corresponding to the DEAR model, (c) shows the result corresponding to the method provided in this embodiment, and (d) shows the result of the attribute intrinsic correlation heatmap predicted by the multimodal large model on a face image. It can be seen that compared with existing models, this model achieves decoupling of fine-grained attributes and perception of relationships between attributes. For example, when the "bangs" attribute increases, the "baldness" attribute shows a negative correlation, while the "blonde hair" attribute shows a positive correlation. This indicates that the model has learned the physical laws of the real world, greatly enhancing the model's practicality.

[0050] See Figure 3 The results on other real image data show that (a) changes in skin color cause changes in the cheekbones, but changes in the cheekbones have little effect on skin color, verifying that the intrinsic relationship of perception is bidirectional and anisotropic. (b) shows that changes in ground color do not affect the horse's running attributes, but when the horse changes from standing to running, it affects the ground color (tending towards sand). (c) shows that the relationship between window size and brightness is also bidirectional and anisotropic. The above verifies that the intrinsic relationship of perceived attributes in this invention is bidirectional and anisotropic.

[0051] In summary, this embodiment achieves fine-grained representation decoupling and association extraction of images, constructing a more practical decoupled perception model. This model combines decoupled representation learning theory with the perception capabilities of a multimodal large model, enabling its application to complex data and real-world scenarios. Simultaneously, the graph-based encoding module provides users with more convenient image control methods; by changing nodes and edges in the graph, corresponding attributes and associations in the image can be controlled, thereby enhancing the model's controllability and generalization. For example, in the 3D human generation task, this embodiment perceives the representational factors of a 3D human, such as the face, arms, legs, and torso, through the attribute factor decoupling module. Simultaneously, it uses a multimodal large model to predict the associations between these attributes and structurally encodes these factors and associations into an explicit graph structure. Based on this explicit graph, the model can generate more realistic 3D humans that conform to physical laws. More specifically, the 3D human's facial features, limb coordination, skin color, and body posture do not exhibit counterfactual generation phenomena, which is of great significance for 3D reconstruction, digital twins, and virtual-real integration.

[0052] In summary, this method has the following beneficial effects:

[0053] (1) This method / device provides a representation decoupling model with the ability to perceive relationships between attributes. Compared with existing methods, this model can perceive and model the relationships between attributes, thus making it suitable for complex data and real-world scenarios.

[0054] (2) This method / device uses the semantic knowledge of large models to predict the intrinsic relationship between representation attributes, and continuously updates and optimizes the network parameters through a graph neural network updater, thereby improving the generalization and robustness of the model.

[0055] (3) This method / device encodes attributes and their inherent relationships in a bidirectional weighted graph, representing attributes and inherent relationships through nodes and edges, respectively. This encoding method enhances the controllability of the model and is beneficial for the practical application of the algorithm.

[0056] (4) It has broad application prospects. Decoupling representation learning can not only improve sample efficiency and reduce sensitivity to interference factors, but also has a wide range of applications in downstream tasks of artificial intelligence.

[0057] Example 2

[0058] This embodiment provides an electronic device, including: one or more processors and a memory, wherein the memory stores one or more programs, the one or more programs including instructions for executing the decoupled representation learning method based on graph structure and multimodal large model as described in Embodiment 1.

[0059] Example 3

[0060] This embodiment provides a computer-readable storage medium including one or more programs executable by one or more processors of an electronic device, the one or more programs including instructions for executing the decoupled representation learning method based on graph structure and multimodal large model as described in Embodiment 1.

[0061] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A decoupled representation learning method based on graph structure and multimodal large model, characterized in that, Includes the following steps: Step S1: Obtain the original image information and perform key point detection; Step S2: Using a preset semantic encoder built on beta-VAE, the key point detection results are mapped to a preset latent space to obtain the attribute factors after fine-grained representation decoupling. In the latent space, each latent code dimension independently represents an attribute or representation. Step S3: Initialize the attribute factors. Based on the key point detection results and the initialized attribute factors, obtain the association relationship between the attribute factors by interacting with the preset large language model. The decoupled representation attributes will be used as the initialization attributes. The preprocessed image and the initialized attribute factors will be input into the attribute association perception module, and interact with the large model through prompting engineering. Step S4: Based on the attribute factors and the relationships between attribute factors, a structured encoding is performed to form a graph structure. In the graph structure, the nodes represent the decoupled attributes, and the edges represent the intrinsic relationships of the attributes predicted by the large model. The edges and weights in the graph structure are updated using a preset graph neural network-based updater. Self-supervised optimization is performed based on the image generated by the decoder and the original image information to achieve decoupled representation learning.

2. The decoupled representation learning method based on graph structure and multimodal large model according to claim 1, characterized in that, The original image information includes a human face image, and the attribute factors include facial features, hair, and skin color.

3. The decoupled representation learning method based on graph structure and multimodal large model according to claim 1, characterized in that, The graph structure includes nodes and edges, where nodes represent fine-grained attribute factors after decoupling, and edges represent the relationships between attribute factors.

4. The decoupled representation learning method based on graph structure and multimodal large model according to claim 1, characterized in that, In step S4, the graph structure is updated using a preset updater based on preset prompts.

5. The decoupled representation learning method based on graph structure and multimodal large model according to claim 1, characterized in that, The graph structure described is a bidirectional weighted graph structure.

6. The decoupled representation learning method based on graph structure and multimodal large model according to claim 1, characterized in that, The updater mentioned includes a graph neural network.

7. A decoupled representation learning device based on graph structure and multimodal large model, characterized in that, include: The attribute factor decoupling module is used to map the key point detection results of the original image information to a preset latent space using a preset semantic encoder based on beta-VAE, so as to obtain the attribute factors after fine-grained representation decoupling. In the latent space, each latent code dimension independently represents an attribute or representation. The attribute association perception module is used to initialize the attribute factors. Based on the key point detection results and the initialized attribute factors, the association relationship between the attribute factors is obtained by interacting with the preset large language model. The decoupled representation attributes will be used as the initialization attributes. The preprocessed image and the initialized attribute factors will be input into the attribute association perception module, and the module will interact with the large model through prompting. The graph structure encoding module is used to form a graph structure by performing structured encoding based on attribute factors and the relationships between attribute factors. In the graph structure, the nodes represent the attributes obtained by decoupling, and the edges represent the intrinsic relationships of attributes predicted by the large model. The edges and weights in the graph structure are updated using a preset graph neural network-based updater. Self-supervised optimization is performed based on the image generated by the decoder and the original image information to achieve decoupled representation learning.

8. An electronic device, characterized in that, include: One or more processors and a memory, wherein the memory stores one or more programs, the one or more programs including instructions for executing the decoupled representation learning method based on graph structure and multimodal large model as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, Includes one or more programs executable by one or more processors of an electronic device, the one or more programs including instructions for executing the decoupled representation learning method based on graph structure and multimodal large model as described in any one of claims 1-6.

Images