A large model driven multi-modal semantic enhancement domain generalization method
By combining MiniCPM, GLM4-0520, and StableDiffusion models with convolutional neural networks and the KMeans algorithm, the problems of large data requirements, high complexity, and difficult fusion in domain generalization methods are solved, achieving efficient multimodal semantic enhancement domain generalization and improving the performance and applicability of the model in new fields.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SOUTHWEST JIAOTONG UNIV
- Filing Date
- 2024-09-18
- Publication Date
- 2026-07-07
AI Technical Summary
Existing domain generalization methods require a large amount of data, which may introduce noise, increase the complexity of data preprocessing and model design, and make it difficult to fuse data from different modalities. In addition, the performance of the model may degrade or the size may become too large in new domains, making it unsuitable for resource-constrained environments.
The MiniCPM model is used to extract semantic information from images, the GLM4-0520 model is used for semantic augmentation, and the StableDiffusion model is combined to generate images. A convolutional neural network and the KMeans algorithm are constructed for clustering and screening. The ResNet-50 backbone network is used for training to achieve multimodal semantic augmentation domain generalization.
It improves the model's generalization ability on data from different domains, reduces data redundancy, and enhances recognition rate and generalization ability. It is applicable to various datasets and has strong adaptability and robustness.
Smart Images
Figure CN119273943B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of machine learning domain generalization technology, and in particular to a large model-driven multimodal semantic enhancement domain generalization method. Background Technology
[0002] Domain generalization aims to improve the generalization ability of machine learning models across different data distributions. Research in this field can be divided into several directions, among which deep learning-based domain generalization methods are currently a highly promising area of research. In deep learning-based domain generalization methods, researchers design various network structures and training strategies to enhance the model's adaptability to data in unknown domains. These methods typically employ supervised learning to learn cross-domain general feature representations on specific structures, and then use the learned general features for inference and prediction in the target domain to achieve domain generalization. Current domain generalization methods mainly include those based on data augmentation, multimodal approaches, and large models.
[0003] While existing domain generalization methods have made some progress, they typically require large amounts of data, making it difficult to guarantee data quality and potentially introducing additional noise. For multimodal data, the need to handle different data types increases the complexity of data preprocessing and model design, and data fusion between different modalities can be challenging. Furthermore, many methods, while improving model generalization ability, sacrifice model performance in new domains or result in excessively large models unsuitable for operation in resource-constrained computing environments. Summary of the Invention
[0004] To address the problems existing in the models of the aforementioned generalization methods, this invention proposes a large model-driven multimodal semantic enhancement domain generalization method to solve the above problems.
[0005] This application discloses a large model-driven multimodal semantic enhancement domain generalization method, including the following steps:
[0006] S1. Dataset acquisition and data domain partitioning.
[0007] S2. Input the dataset into the MiniCPM model to extract the semantic information of the images. The MiniCPM model can efficiently extract rich semantic information from the input images, including key features and stylistic descriptions of the images.
[0008] S3. Each key feature word in the extracted semantic information is semantically augmented using the GLM4-0520 model, and the combined results are a detailed semantic description. The GLM4-0520 model is used to augment each piece of semantic information extracted from step S2.
[0009] S4. The detailed semantic description is used to generate the corresponding image using the StableDiffusion model. The StableDiffusion model uses the detailed semantic description obtained in step S3 as input.
[0010] S5. Construct a convolutional neural network and combine it with the KMeans algorithm to perform clustering and filtering on the images generated in S4, retaining one image for each cluster.
[0011] S6. Merge the images selected in S5 with the dataset in S1 to obtain the merged dataset, and input it into the ResNet-50 backbone network for training to obtain the trained model. The ResNet-50 backbone network is used to improve the speed and accuracy of feature extraction.
[0012] S7. Input the test domain data into the model trained in S6 to test the model performance.
[0013] Preferably, the dataset is the VLCS dataset, a cross-domain image classification dataset containing 10,729 images from four subsets: Caltech101, LabelMe, SUN09, and VOC2007. The different subsets differ in data distribution and image size, but all contain five categories: birds, cars, chairs, dogs, and people.
[0014] Preferably, the data domain partitioning includes the following steps:
[0015] The VLCS dataset is divided into four sub-datasets: a training domain and a test domain. One or more datasets are selected as the training domain to train the model, and the remaining data are used as the test domain to evaluate the model's performance.
[0016] Preferably, the semantic information extracted in S2 includes at least three key image feature words and at least one style description word.
[0017] Preferably, the convolutional neural network includes a first convolutional block, a second convolutional block, a third convolutional block, and a fourth convolutional block connected in sequence. Each convolutional block does not have a fully connected layer, and each convolutional block includes a convolutional layer, a normalization layer, an activation function, and a pooling layer.
[0018] Preferably, the first convolutional block includes a convolutional layer C1_1, a normalization layer B1_1, an activation function S1_1, a convolutional layer C1_2, a normalization layer B1_2, an activation function S1_2, and a pooling layer M1;
[0019] The second convolutional block includes a convolutional layer C2_1, a normalization layer B2_1, an activation function S2_1, a convolutional layer C2_2, a normalization layer B2_2, an activation function S2_2, and a pooling layer M2;
[0020] The third convolutional block includes a convolutional layer C3_1, a normalization layer B3_1, an activation function S3_1, a convolutional layer C3_2, a normalization layer B3_2, an activation function S3_2, a convolutional layer C3_3, a normalization layer B3_3, an activation function S3_3, and a pooling layer M3.
[0021] The fourth convolutional block includes a convolutional layer C4_1, a normalization layer B4_1, an activation function S4_1, a convolutional layer C4_2, a normalization layer B4_2, an activation function S4_2, a convolutional layer C4_3, a normalization layer B4_3, an activation function S4_3, and a pooling layer M4.
[0022] Preferably, the kernel size in each convolutional block is 3*3, and the stride is 1.
[0023] The pooling layers in each convolutional block use the max pooling method, with a pooling window size of 2*2 and a stride of 2.
[0024] Preferably, the activation function in each convolutional block is the SeLU function, and its calculation formula is as follows:
[0025]
[0026] Where x is the data to be processed, f(x) is the processed result, and λ and α are two constants.
[0027] Preferably, the S5 step of performing clustering and filtering on the image generated in S4 using the KMeans algorithm includes the following steps:
[0028] Image feature extraction involves extracting features from an image that can describe its content and representing them as vectors.
[0029] The Euclidean distance is used as a metric to evaluate the similarity between vectors. The formula for calculating the Euclidean distance is:
[0030]
[0031] Where A and B are any two vectors, a i Let b be the i-th coordinate of vector A. i Let be the i-th coordinate of vector B.
[0032] Preferably, step S6 includes the following steps:
[0033] The images filtered in S5 are merged with the dataset used as the training domain in S1. The training domain data of the same category and the filtered images are put together and used as training data for the backbone network.
[0034] The beneficial effects of this invention are:
[0035] (1) This invention addresses the domain generalization problem by using semantic enhancement technology to replace traditional image enhancement methods, thereby integrating multimodal data into the domain generalization framework and solving the problem that traditional enhancement methods such as scaling and cropping cannot expand feature diversity.
[0036] (2) This invention introduces the MiniCPM model, GLM4-0520 model, StableDiffusion model and other models into domain generalization research, and utilizes their advantages in cross-domain knowledge transfer to help improve the generalization ability of the model on data in different domains.
[0037] (3) This invention uses convolutional neural networks and KMeans clustering algorithm to cluster and filter images. By selecting a representative image from each class, data redundancy can be effectively reduced, while improving the generalization ability of the model.
[0038] (4) The model of the present invention has a high correct recognition rate and generalization ability, which can provide strong support for solving the challenges in cross-domain learning. It can be deployed in various domain generalization application scenarios. It can not only achieve a high recognition accuracy on known training domains, but also has no restrictions on the selection of datasets. Therefore, it can be applied to various types of datasets and has a wide range of applicability. Attached Figure Description
[0039] Figure 1 This is a flowchart of the large model-driven multimodal semantic enhancement domain generalization method according to an embodiment of the present invention;
[0040] Figure 2 This is a schematic diagram of the convolutional neural network structure according to an embodiment of the present invention;
[0041] Figure 3 This is a schematic diagram illustrating data changes using actual images, representing an embodiment of the present invention. Detailed Implementation
[0042] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided with reference to the accompanying drawings and embodiments.
[0043] This application discloses a large model-driven multimodal semantic enhancement domain generalization method, such as... Figure 1 As shown, it includes the following steps:
[0044] S1. Dataset acquisition and data domain partitioning.
[0045] In this implementation, the VLCS dataset was chosen. The VLCS dataset is a cross-domain image classification dataset containing 10,729 images from four subsets: Caltech101, LabelMe, SUN09, and VOC2007. The different subsets differ in data distribution and image size, but all contain five categories: birds, cars, chairs, dogs, and people.
[0046] The VLCS dataset is divided into four sub-datasets: a training domain and a test domain. One or more datasets are selected as the training domain to train the model, and the remaining data are used as the test domain to evaluate the model's performance.
[0047] S2. Input the dataset into the MiniCPM model to extract the semantic information of the images.
[0048] The MiniCPM model is a multimodal model pre-trained on large-scale image and text data. It can efficiently extract rich semantic information from input images, including key image features and stylistic descriptions. The MiniCPM model is used to read input images and extract their semantic information, which includes at least three key feature words and one stylistic description word.
[0049] The MiniCPM model is guided by constructed prompts to read the input image and return semantic information. The prompts used are: "describe the main content of the image briefly" and "What is the style of the image (egoil painting, photography, simple drawing)".
[0050] Extracting rich semantic information in the above way helps to understand the image content more accurately, and can also improve the adaptability and robustness of the model in the process of cross-domain generalization.
[0051] S3. Each key feature word in the extracted semantic information is semantically augmented using the GLM4-0520 model, and then combined (i.e., the augmented key feature words are spliced and permuted) to obtain a detailed semantic description.
[0052] The GLM4-0520 model, as a powerful natural language processing model, is used to augment each semantic information extracted from step S2. The GLM4-0520 model can understand the deeper meaning of each feature in this semantic information and expand it to generate feature descriptions that are similar but different in meaning.
[0053] The GLM4-0520 model is guided by constructed prompts to expand upon the features in the semantic information, generating feature descriptions that are similar to but different in meaning. The prompts used are: "As an AI assistant, you now need to read and understand a description of an image. Simplify it, and consider all possible things mentioned in the description to be correct. The extracted information should include at least: the style of the image, a description of the main information in the image, and a description of the background of the image." "Further, simplify the above information, generate four adjectives or phrases (ensuring that one of them is related to the art style), and expand each of these four adjectives separately, requiring each adjective to be expanded with three extension words or phrases of the same category but different attributes. For example, 'watercolor style' can be extended to include simple strokes, oil paintings, and photographs."The output only includes a list of 4adjectives and their expanded results,with all expanded words for eachadjective occupying 1line,separated by commas","Return a 4-line stringwrapped in the format of",with each line starting with the extracteddescriptor and followed by 3additional descriptors separated by commas.Noother content needs to be answered.For example:"'Simplistic:basic design, plain lines,unadorned form\nPink:soft pink,bright fuchsia,pastel rose\nArtistic:sculptural piece,illustrative drawing,artistic rendering\nStylized:abstract form,cartoonish,simplified shape”.
[0054] In the above manner, each semantic information obtained is amplified, and these features are combined to form new semantic descriptions, further enriching the semantic content of the image and providing more comprehensive guidance for subsequent image generation.
[0055] S4. Use the StableDiffusion model to generate the corresponding image from the detailed semantic description.
[0056] The StableDiffusion model uses the detailed semantic descriptions obtained in step S3 as input. These semantic descriptions not only include key features of the image, but may also include higher-level semantic information such as style descriptions.
[0057] When the StableDiffusion model generates images, prompts need to be constructed to guide the image generation, thereby improving image quality and preventing the image from losing its primary focus. The prompts consist of image semantics and additional image generation requirements. The image semantics are the semantic information generated in S3, and the additional image generation requirements include "close-up," meaning setting the image composition to a medium foreground / midground, while simultaneously setting the weight of the word representing the main subject to 3.
[0058] Through the above approach, the entire process realizes the transformation from input image to detailed semantic description, and then to the generation and conversion of specific images. This domain generalization method based on multimodal semantic enhancement not only improves the model's generalization ability under different data distributions, but also enhances the model's understanding and expression of image content.
[0059] S5. Construct a convolutional neural network and combine it with the KMeans algorithm to perform clustering and filtering on the images generated in S4, retaining one image for each cluster. The KMeans algorithm divides the images into 20 classes by calculating the similarity between image features, with images within each class having high visual similarity. By selecting a representative image from each class, data redundancy can be effectively reduced while preserving the uniqueness of images from different categories.
[0060] like Figure 2 As shown, the convolutional neural network used for feature extraction includes a first convolutional block, a second convolutional block, a third convolutional block, and a fourth convolutional block connected in sequence. The convolutional blocks do not have fully connected layers. Each convolutional block includes a convolutional layer, a normalization layer, an activation function, and a pooling layer.
[0061] The first convolutional block includes a convolutional layer C1_1, a normalization layer B1_1, an activation function S1_1, a convolutional layer C1_2, a normalization layer B1_2, an activation function S1_2, and a pooling layer M1;
[0062] The second convolutional block includes a convolutional layer C2_1, a normalization layer B2_1, an activation function S2_1, a convolutional layer C2_2, a normalization layer B2_2, an activation function S2_2, and a pooling layer M2;
[0063] The third convolutional block includes convolutional layer C3_1, normalization layer B3_1, activation function S3_1, convolutional layer C3_2, normalization layer B3_2, activation function S3_2, convolutional layer C3_3, normalization layer B3_3, activation function S3_3, and pooling layer M3;
[0064] The fourth convolutional block includes convolutional layer C4_1, normalization layer B4_1, activation function S4_1, convolutional layer C4_2, normalization layer B4_2, activation function S4_2, convolutional layer C4_3, normalization layer B4_3, activation function S4_3, and pooling layer M4.
[0065] The kernel size in each convolutional block is 3*3, and the stride is 1. The pooling layer in each convolutional block uses the max pooling method, with the pooling window size set to 2*2 and the stride to 2.
[0066] The activation function in each convolutional block is the SeLU function, and its calculation formula is as follows:
[0067]
[0068] Where x is the data to be processed, f(x) is the processed result, and λ and α are two constants. In this embodiment, λ = 1.0507 and α = 1.67326.
[0069] When applied to image data, the KMeans clustering algorithm first extracts features from the image, that is, it extracts features that describe the image content and represents them in vector form. Then, to evaluate the similarity between these vectors, Euclidean distance is used as a metric. The formula for calculating Euclidean distance is as follows:
[0070]
[0071] Where A and B are any two vectors, a i Let b be the i-th coordinate of vector A. i Let be the i-th coordinate of vector B.
[0072] Through the methods described above, the KMeans clustering algorithm demonstrates its unique advantages in image processing, particularly in organizing and classifying large numbers of generated images. By clustering, the algorithm not only effectively organizes images, grouping visually similar images together, but also significantly reduces data redundancy by selecting representative images while preserving data diversity.
[0073] S6. Merge the images selected in S5 with the dataset in S1. Combine the training domain data of the same category with the selected images, and use the combined dataset as training data for the backbone network. Input the merged dataset into the ResNet-50 backbone network for training to obtain the trained model. The ResNet-50 backbone network is used to improve the speed and accuracy of feature extraction. In this embodiment, the training parameters are set as follows: initial learning rate is 0.001, batch size is 64, SGD optimizer is used, and momentum is 0.9. The entire training process is divided into 50 rounds. After each round, the accuracy is used as the performance metric, and the learning rate or training strategy is adjusted based on the evaluation results. Output the accuracy obtained after each round of training, and finally save the trained model. Through the above steps, the model's performance in new domains can be effectively improved, enhancing its generalization ability and robustness.
[0074] S7. Input the test domain data into the model trained in S6 to test the model performance.
[0075] The test domain data obtained in S1 is input into the backbone model trained in S6 for classification to test model performance. The test domain differs from the training domain and is used to test the model's domain generalization ability to accurately measure the model's performance in cross-domain situations.
[0076] Performance is evaluated using test domain data, and the model's generalization performance is measured by its accuracy. The model effectively utilizes the trained data on the test domain, demonstrating its adaptability and generalization level.
[0077] In a specific embodiment, such as Figure 3 As shown, firstly, an image from the training domain is input. The MiniCPM model is used to obtain semantic information from the image data, such as "The image features a pink flamingo with a white beak and black eye. The image appears to be a simple drawing or sculpture of a flamingo.|bird". Then, the GLM4-0520 model is used to semantically amplify the key features extracted from the semantic information. After combining the amplified features, several detailed semantic descriptions are obtained, such as:
[0078] “sharp clarity,balanced pose,serene pond,bird
[0079] sharp clarity,balanced pose,reflective surface,bird
[0080] sharp clarity,balanced pose,tranquil setting,bird
[0081] sharp clarity,standing tall,serene pond,bird
[0082] sharp clarity,standing tall,reflective surface,bird
[0083] (Clear, balanced posture, tranquil pond, birds)
[0084] Clear, balanced posture, reflective surface, bird
[0085] A clear, balanced posture, a tranquil environment, and birds.
[0086] Clear, towering, tranquil pond, birds
[0087] (Clear, upright, reflective surface, bird). All detailed descriptions are input into the StableDiffusion model to generate corresponding images. Convolutional neural networks are used to extract image features, and KMeans clustering algorithm is used to cluster the images, resulting in 20 clusters. The center image of each cluster is selected and merged with the training domain image data, and then put into the backbone network for training. Finally, the test domain data is input into the trained model to test the model performance.
[0088] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.
Claims
1. A large model-driven multimodal semantic enhancement domain generalization method, characterized in that, Includes the following steps: S1. Dataset acquisition and data domain partitioning; S2. Input the dataset into the MiniCPM model and extract the semantic information of the images; including: constructing prompt words to guide the MiniCPM model to read the input images and return semantic information; S3. Each key feature word in the extracted semantic information is semantically augmented using the GLM4-0520 model, and the resulting combination yields a detailed semantic description. This includes: guiding the GLM4-0520 model by constructing prompt words to expand each feature in the semantic information and generate feature descriptions that are similar to but different in meaning. S4. Use the StableDiffusion model to generate corresponding images based on detailed semantic descriptions; including: when the StableDiffusion model generates images, construct prompt words to guide it, so as to improve the quality of the generated images and avoid the image being unclear. The prompt words are composed of image semantics and additional generated image requirements. S5. Construct a convolutional neural network and combine it with the KMeans algorithm to perform clustering and filtering on the images generated in S4, retaining one image for each cluster. S6. Merge the images selected in S5 with the dataset in S1 to obtain the merged dataset and input it into the backbone network ResNet-50 for training to obtain the trained model. S7. Input the test domain data into the model trained in S6 to test the model performance.
2. The large model-driven multimodal semantic enhancement domain generalization method according to claim 1, characterized in that, The dataset in question is the VLCS dataset, which includes four sub-datasets: Caltech101, LabelMe, SUN09, and VOC2007. The different sub-datasets differ in data distribution and image size, but all contain five categories: birds, cars, chairs, dogs, and people.
3. The large model-driven multimodal semantic enhancement domain generalization method according to claim 2, characterized in that, The data domain partitioning includes the following steps: The VLCS dataset is divided into four sub-datasets: a training domain and a test domain. One or more datasets are selected as the training domain to train the model, and the remaining data are used as the test domain to evaluate the model's performance.
4. The large model-driven multimodal semantic enhancement domain generalization method according to claim 3, characterized in that, The semantic information extracted in S2 includes at least three key feature words and at least one descriptive word for the art style.
5. The large model-driven multimodal semantic enhancement domain generalization method according to claim 4, characterized in that, The convolutional neural network includes a first convolutional block, a second convolutional block, a third convolutional block, and a fourth convolutional block connected in sequence. Each convolutional block does not have a fully connected layer. Each convolutional block includes a convolutional layer, a normalization layer, an activation function, and a pooling layer.
6. The large model-driven multimodal semantic enhancement domain generalization method according to claim 5, characterized in that, The first convolutional block includes a convolutional layer C1_1, a normalization layer B1_1, an activation function S1_1, a convolutional layer C1_2, a normalization layer B1_2, an activation function S1_2, and a pooling layer M1; The second convolutional block includes a convolutional layer C2_1, a normalization layer B2_1, an activation function S2_1, a convolutional layer C2_2, a normalization layer B2_2, an activation function S2_2, and a pooling layer M2; The third convolutional block includes a convolutional layer C3_1, a normalization layer B3_1, an activation function S3_1, a convolutional layer C3_2, a normalization layer B3_2, an activation function S3_2, a convolutional layer C3_3, a normalization layer B3_3, an activation function S3_3, and a pooling layer M3. The fourth convolutional block includes a convolutional layer C4_1, a normalization layer B4_1, an activation function S4_1, a convolutional layer C4_2, a normalization layer B4_2, an activation function S4_2, a convolutional layer C4_3, a normalization layer B4_3, an activation function S4_3, and a pooling layer M4.
7. The large model-driven multimodal semantic enhancement domain generalization method according to claim 6, characterized in that, The kernel size in each convolutional block is 3*3, and the stride is 1. The pooling layers in each convolutional block use the max pooling method, with a pooling window size of 2*2 and a stride of 2.
8. The large model-driven multimodal semantic enhancement domain generalization method according to claim 7, characterized in that, The activation function in each convolutional block is the SeLU function, and its calculation formula is as follows: in, For the data to be processed, The result after processing. and These are two constants.
9. The large model-driven multimodal semantic enhancement domain generalization method according to claim 8, characterized in that, The S5 step, which combines the KMeans algorithm to perform clustering and filtering on the image generated in S4, includes the following steps: Image feature extraction involves extracting features from an image that can describe its content and representing them as vectors. The Euclidean distance is used as a metric to evaluate the similarity between vectors. The formula for calculating the Euclidean distance is: in, and For any two vectors, For vectors The coordinates, For vectors The Coordinates.
10. The large model-driven multimodal semantic enhancement domain generalization method according to claim 9, characterized in that, S6 includes the following steps: The images filtered in S5 are merged with the dataset used as the training domain in S1. The training domain data of the same category and the filtered images are put together and used as training data for the backbone network.