Large-model-driven multi-modal semantic enhancement domain generalization method

Abstract

The invention discloses a multi-modal semantic enhancement domain generalization method driven by a large model. The method comprises the following steps: S1, acquiring a data set and dividing a data domain; s2, inputting the data set into a Mini CPM model, and extracting semantic information of the image; s3, semantic amplification is carried out through a GLM4-0520 model, and detailed semantic description is obtained after combination; s4, generating a corresponding image from the detailed semantic description by using a Stable Diffusion model; s5, constructing a convolutional neural network, and performing clustering screening processing on the image in combination with a KMeans algorithm; s6, combining the screened images with the data set to obtain a combined data set, and inputting the combined data set into a backbone network Resnet-50 for training; and S7, inputting the test domain data into the model trained in the step S6, and testing the performance of the model. According to the method, multiple models are introduced, and the generalization ability and accuracy of data in different fields are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of machine learning domain generalization, and in particular to a large model-driven multimodal semantically enhanced domain generalization method. Background Art

[0002] Domain generalization aims to improve the generalization ability of machine learning models under different data distributions. Research in this field can be divided into multiple directions, among which domain generalization methods based on deep learning are one of the research directions that have attracted much attention. In domain generalization methods based on deep learning, researchers design various network structures and training strategies to enhance the model's adaptability to unknown domain data. These methods usually use supervised learning methods to learn cross-domain universal feature representations on specific structures, and then use the learned universal features to perform reasoning and prediction on the target domain to achieve domain generalization. The existing domain generalization methods mainly include methods based on data enhancement, multimodality, and large models.

[0003] Although existing domain generalization methods have made some progress, they usually require a large amount of data, which makes it difficult to ensure data quality and may introduce additional noise. For multimodal data, the need to process different types of data increases the complexity of data preprocessing and model design, and data fusion between different modalities may be difficult. In addition, many methods improve the generalization ability of the model while sacrificing the performance of the model in new domains, or causing the model to be too large and unsuitable for running in a computing environment with limited resources. Summary of the invention

[0004] In order to solve the problems existing in the models of the above-mentioned existing generalization methods, the present invention proposes a large model-driven multimodal semantic enhancement domain generalization method to solve the above-mentioned problems.

[0005] The present application discloses a large model driven multimodal semantic enhancement domain generalization method, comprising the following steps:

[0006] S1. Data set acquisition and data domain division.

[0007] S2. Input the dataset into the MiniCPM model to extract the semantic information of the image. The MiniCPM model can efficiently extract rich semantic information from the input image, including the key features and style description of the image.

[0008] S3, semantically amplify each key feature word in the extracted semantic information through the GLM4-0520 model, and obtain a detailed semantic description after combination. The GLM4-0520 model is used to amplify each semantic information extracted from step S2.

[0009] S4. Generate the corresponding image using the StableDiffusion model. The StableDiffusion model uses the detailed semantic description obtained in step S3 as input.

[0010] S5. Construct a convolutional neural network and use the KMeans algorithm to perform clustering and screening on the images generated by S4, retaining one image for each cluster.

[0011] S6, merge the image screened by S5 with the data set in S1, obtain a merged data set and input it into the backbone network Resnet-50 for training to obtain a trained model. The backbone network Resnet-50 is used to improve the speed and accuracy of feature extraction.

[0012] S7. Input the test domain data into the model trained in S6 to test the model performance.

[0013] Preferably, the dataset is a VLCS dataset, which is a cross-domain image classification dataset that contains 10,729 images from four sub-datasets: Caltech101, LabelMe, SUN09, and VOC2007. Different sub-datasets differ in data distribution and image size, but all contain five categories, namely, bird, car, chair, dog, and person.

[0014] Preferably, the data domain division comprises the following steps:

[0015] The four sub-datasets in the VLCS dataset are divided into training domains and test domains. Each time, one or more datasets are selected as training domains to train the model, and the remaining data are used as test domains to evaluate the performance of the model.

[0016] Preferably, the semantic information extracted in S2 includes at least three key image feature words and at least one painting style description word.

[0017] Preferably, the convolutional neural network includes a first convolution block, a second convolution block, a third convolution block and a fourth convolution block connected in series in sequence, the convolution block is not provided with a fully connected layer, and each convolution block includes a convolution layer, a normalization layer, an activation function and a pooling layer.

[0018] Preferably, the first convolution block includes a convolution layer C1_1, a normalization layer B1_1, an activation function S1_1, a convolution layer C1_2, a normalization layer B1_2, an activation function S1_2 and a pooling layer M1;

[0019] The second convolution block includes a convolution layer C2_1, a normalization layer B2_1, an activation function S2_1, a convolution layer C2_2, a normalization layer B2_2, an activation function S2_2 and a pooling layer M2;

[0020] The third convolution block includes a convolution layer C3_1, a normalization layer B3_1, an activation function S3_1, a convolution layer C3_2, a normalization layer B3_2, an activation function S3_2, a convolution layer C3_3, a normalization layer B3_3, an activation function S3_3 and a pooling layer M3;

[0021] The fourth convolution block includes a convolution layer C4_1, a normalization layer B4_1, an activation function S4_1, a convolution layer C4_2, a normalization layer B4_2, an activation function S4_2, a convolution layer C4_3, a normalization layer B4_3, an activation function S4_3 and a pooling layer M4.

[0022] Preferably, the convolution kernel size in each convolution block is 3*3, and the step size is 1;

[0023] The pooling layer in each convolution block uses the maximum pooling method, and the pooling window size is set to 2*2 and the step size is 2.

[0024] Preferably, the activation function in each convolution block is a SeLU function, and its calculation formula is:

[0025]

[0026] Among them, x is the data to be processed, f(x) is the result after processing, and λ and α are two constants.

[0027] Preferably, the clustering screening process of the image generated by S4 in combination with the KMeans algorithm in S5 includes the following steps:

[0028] Perform feature extraction on the image, that is, extract features that can describe the image content from the image and express them in the form of vectors;

[0029] The Euclidean distance is used as a metric to evaluate the similarity between vectors. The calculation formula of the Euclidean distance is:

[0030]

[0031] Among them, A and B are any two vectors, a i is the i-th coordinate of vector A, b i is the i-th coordinate of vector B.

[0032] Preferably, S6 comprises the following steps:

[0033] The filtered images in S5 are merged with the dataset used as the training domain in S1. The training domain data and filtered images of the same category are put together and used as the training data for the backbone network as a whole.

[0034] Beneficial effects of the present invention:

[0035] (1) The present invention aims at the domain generalization problem. By replacing the traditional image enhancement method with semantic enhancement technology, the multimodal data is integrated into the domain generalization framework, which solves the problem that traditional enhancement methods such as scaling and cropping cannot expand feature diversity.

[0036] (2) The present invention introduces the MiniCPM model, GLM4-0520 model, StableDiffusion model, etc. into the domain generalization research, and utilizes their advantages in cross-domain knowledge transfer, which helps to improve the generalization ability of the model on data in different fields.

[0037] (3) The present invention uses convolutional neural networks and KMeans clustering algorithms to implement clustering screening of images. By selecting a representative image in each class, the redundancy of data can be effectively reduced while improving the generalization ability of the model.

[0038] (4) The model of the present invention has a high correct recognition rate and generalization ability, which can provide strong support for solving the challenges in cross-domain learning and can be deployed in various domain generalization application scenarios. It can not only achieve a high recognition accuracy rate on a known training domain, but also has no restrictions on the choice of data sets. Therefore, it can be applied to various types of data sets and has a wide range of applicability. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a flow chart of a multimodal semantic enhancement domain generalization method driven by a large model according to an embodiment of the present invention;

[0040] Figure 2 Schematic diagram of the convolutional neural network structure of an embodiment of the present invention;

[0041] Figure 3 It is a schematic diagram of data changes displayed in actual images according to an embodiment of the present invention. DETAILED DESCRIPTION

[0042] In order to make the objectives, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below with reference to the accompanying drawings and examples.

[0043] The present application embodiment discloses a large model driven multimodal semantic enhancement domain generalization method, such as Figure 1 As shown, the following steps are included:

[0044] S1. Data set acquisition and data domain division.

[0045] In this implementation, the dataset selected is the VLCS dataset, which is a cross-domain image classification dataset that contains 10,729 images from four sub-datasets: Caltech101, LabelMe, SUN09, and VOC2007. Different sub-datasets differ in data distribution and image size, but all contain five categories: bird, car, chair, dog, and person.

[0046] The four sub-datasets in the VLCS dataset are divided into training domains and test domains. Each time, one or more datasets are selected as training domains to train the model, and the remaining data are used as test domains to evaluate the performance of the model.

[0047] S2. Input the dataset into the MiniCPM model to extract the semantic information of the image.

[0048] The MiniCPM model is a multimodal model pre-trained based on large-scale image and text data. It can efficiently extract rich semantic information from the input image, including the key features and style description of the image. The MiniCPM model is used to read the input image and extract the semantic information of the image, which includes at least three key feature words and one style description word.

[0049] Construct prompt words to guide the MiniCPM model to read the input image and return semantic information. The prompt words used are: "describe the main content of the image briefly", "What is the style of the image (egoil painting, photography, simple drawing)".

[0050] Extracting rich semantic information in the above way helps to understand the image content more accurately and improve the adaptability and robustness of the model in the cross-domain generalization process.

[0051] S3. Each key feature word in the extracted semantic information is semantically amplified through the GLM4-0520 model, and a detailed semantic description is obtained after combination (i.e., splicing and fully arranging the amplified key feature words).

[0052] As a powerful natural language processing model, the GLM4-0520 model is used to amplify each semantic information extracted from step S2. The GLM4-0520 model can understand the deep meaning of each feature in these semantic information and expand it to generate similar but different feature descriptions.

[0053] The GLM4-0520 model is guided by constructing prompt words to expand each feature in the semantic information and generate feature descriptions of the same category but different meanings. The prompt words used are: "As an AI assistant, you now need to read and understand a description of an image, simplify it, and consider all possible things mentioned in the description to be correct. The extracted information should include at least: the style of the image, description of the main information in the image, and a description of the background of the image", "Simplify the above information further, generate 4 adjectives or phrases (ensuring that 1 of them is related to the art style), and expand each of these 4 adjectives separately, requiring each adjective to be expanded with 3 extension words or phrases of the same category but different attributes. For example, watercolor style can be extended to includes simple strokes, oil paintings, and photographs.The output only includes a list of 4adjectives and their expanded results,with all expanded words for eachadjective occupying 1line,separated by commas","Return a 4-line stringwrapped in the format of",with each line starting with the extracteddescriptor and followed by 3additional descriptors separated by commas.Noother content needs to be answered.For example:"'Simplistic:basic design, plain lines,unadorned form\nPink:soft pink,bright fuchsia,pastel rose\nArtistic:sculptural piece,illustrative drawing,artistic rendering\nStylized:abstract form,cartoonish,simplified shape”.

[0054] Through the above method, each semantic information obtained is amplified, and these features are combined with each other to form a new semantic description, which further enriches the semantic content of the image and provides more comprehensive guidance for subsequent image generation.

[0055] S4. Use the StableDiffusion model to generate the corresponding image using the detailed semantic description.

[0056] The StableDiffusion model uses the detailed semantic descriptions obtained in step S3 as input. These semantic descriptions not only contain the key features of the image, but may also include more advanced semantic information such as style description.

[0057] When the StableDiffusion model generates images, it is necessary to construct prompt words to guide them in order to improve the quality of the raw images and avoid confusing the main and secondary images. The prompt words are composed of image semantics and additional raw image requirements. The image semantics is the semantic information generated in S3. The additional raw image requirements add "close-up", which means setting the image composition to a medium close-up, and setting the weight of the words representing the subject to 3.

[0058] Through the above method, the entire process realizes the generation and conversion from input image to detailed semantic description and then to specific image. This domain generalization method based on multimodal semantic enhancement not only improves the generalization ability of the model under different data distributions, but also enhances the model's ability to understand and express image content.

[0059] S5, construct a convolutional neural network, and use the KMeans algorithm to perform clustering and screening on the images generated by S4, retaining one image in each cluster. The KMeans algorithm calculates the similarity between image features and divides them into 20 classes. The images in each class have high visual similarity. By selecting a representative image in each class, the redundancy of the data can be effectively reduced while retaining the uniqueness of images of different categories.

[0060] like Figure 2 As shown, the convolutional neural network for extracting features includes a first convolution block, a second convolution block, a third convolution block and a fourth convolution block connected in series in sequence. The convolution block does not have a fully connected layer, and each convolution block includes a convolution layer, a normalization layer, an activation function and a pooling layer.

[0061] The first convolution block includes convolution layer C1_1, normalization layer B1_1, activation function S1_1, convolution layer C1_2, normalization layer B1_2, activation function S1_2 and pooling layer M1;

[0062] The second convolution block includes a convolution layer C2_1, a normalization layer B2_1, an activation function S2_1, a convolution layer C2_2, a normalization layer B2_2, an activation function S2_2, and a pooling layer M2;

[0063] The third convolution block includes a convolution layer C3_1, a normalization layer B3_1, an activation function S3_1, a convolution layer C3_2, a normalization layer B3_2, an activation function S3_2, a convolution layer C3_3, a normalization layer B3_3, an activation function S3_3 and a pooling layer M3;

[0064] The fourth convolution block includes a convolution layer C4_1, a normalization layer B4_1, an activation function S4_1, a convolution layer C4_2, a normalization layer B4_2, an activation function S4_2, a convolution layer C4_3, a normalization layer B4_3, an activation function S4_3 and a pooling layer M4.

[0065] The convolution kernel size in each convolution block is 3*3, and the step size is 1; the pooling layer in each convolution block uses the maximum pooling method, setting the pooling window size to 2*2 and the step size to 2.

[0066] The activation function in each convolution block is the SeLU function, and its calculation formula is:

[0067]

[0068] Wherein, x is the data to be processed, f(x) is the result after processing, and λ and α are two constants. In this embodiment, λ=1.0507, α=1.67326.

[0069] When the KMeans clustering algorithm is applied to image data, it first extracts features from the image, that is, extracts features that can describe the image content from the image and represents them in the form of vectors. Then, in order to evaluate the similarity between these vectors, the Euclidean distance is used as a metric. The calculation formula of the Euclidean distance is as follows:

[0070]

[0071] Among them, A and B are any two vectors, a i is the i-th coordinate of vector A, b i is the i-th coordinate of vector B.

[0072] In the above way, the application of KMeans clustering algorithm in image processing, especially in organizing and classifying a large number of generated images, shows its unique advantages. Through clustering, the algorithm not only realizes the effective organization of images and classifies visually similar images into one category, but also significantly reduces the redundancy of data by selecting representative images, while retaining the diversity of data.

[0073] S6, merge the image obtained by S5 screening with the data set in S1, put the training domain data of the same category and the screened image together, and use them as training data for the backbone network as a whole. The data set is merged and input into the backbone network Resnet-50 for training to obtain a trained model. The backbone network Resnet-50 is used to improve the speed and accuracy of feature extraction. In this embodiment, the training parameters are set as follows: the initial learning rate is set to 0.001, batch_size is set to 64, the optimizer uses SGD, and momentum is set to 0.9. The entire training process is divided into 50 rounds, and the performance is evaluated with the accuracy rate as an indicator after each round of training, and the learning rate or training strategy is adjusted according to the evaluation results. The accuracy rate obtained after each round of training is output, and the training model is finally saved. Through the above steps, the performance of the model in the new field can be effectively improved, and the generalization ability and robustness of the model can be improved.

[0074] S7. Input the test domain data into the model trained in S6 to test the model performance.

[0075] The test domain data obtained in S1 is input into the backbone model trained in S6 for classification to test the model performance. The test domain is different from the training domain and is used to test the domain generalization ability of the model to accurately measure the performance of the model in cross-domain situations.

[0076] Use the test domain data for performance evaluation, and measure the generalization performance of the model based on the model accuracy. The trained model can be effectively used on the test domain data, demonstrating its adaptability and generalization level to the test domain data.

[0077] In a specific embodiment, Figure 3 As shown in the figure, firstly, an image data from the training domain is input, and the semantic information of the image data is obtained through the MiniCPM model, such as "The image features a pink flamingo with awhite beak and black eye. The image appears to be a simple drawing or sculpture of a flamingo.|bird". Then, the key features extracted from the semantic information are semantically amplified through the GLM4-0520 model, and the amplified features are combined to obtain multiple detailed semantic descriptions, such as "sharp clarity, balanced pose, serene pond, bird

[0078] sharp clarity,balanced pose,reflective surface,bird

[0079] sharp clarity,balanced pose,tranquil setting,bird

[0080] sharp clarity,standing tall,serene pond,bird

[0081] All detailed descriptions are input into the StableDiffusion model to generate corresponding images. The convolutional neural network is used to extract image features and the KMeans clustering algorithm is used to cluster the images. Twenty clusters are obtained by clustering. The central image of each cluster is screened out and merged with the training domain image data and put into the backbone network for training. Finally, the test domain data is input into the trained model to test the model performance.

[0082] The above shows and describes the basic principles, main features and advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A large model driven multimodal semantic enhancement domain generalization method, characterized in that: The following steps are involved: S1, data set acquisition and data domain division; S2, input the data set into the MiniCPM model to extract the semantic information of the image; S3, semantically amplify each key feature word in the extracted semantic information through the GLM4-0520 model, and obtain a detailed semantic description after combination; S4, using the StableDiffusion model to generate the corresponding image using the detailed semantic description; S5, construct a convolutional neural network, and use the KMeans algorithm to perform clustering and screening on the images generated by S4, retaining one image for each cluster; S6, merge the image screened by S5 with the data set in S1 to obtain a merged data set and input it into the backbone network Resnet-50 for training to obtain a trained model; S7. Input the test domain data into the model trained in S6 to test the model performance.

2. The large model driven multimodal semantic enhancement domain generalization method according to claim 1, characterized in that: The dataset is the VLCS dataset, which includes four sub-datasets: Caltech101, LabelMe, SUN09, and VOC2007. Different sub-datasets differ in data distribution and image size, but all contain five categories, namely, bird, car, chair, dog, and person.

3. The large model driven multimodal semantic enhancement domain generalization method according to claim 2, characterized in that: The data domain division comprises the following steps: The four sub-datasets in the VLCS dataset are divided into training domains and test domains. Each time, one or more datasets are selected as training domains to train the model, and the remaining data are used as test domains to evaluate the performance of the model.

4. The large model driven multimodal semantic enhancement domain generalization method according to claim 3, characterized in that: The semantic information extracted in S2 includes at least three key feature words and at least one painting style description word.

5. The large model driven multimodal semantic enhancement domain generalization method according to claim 4, characterized in that: The convolutional neural network includes a first convolution block, a second convolution block, a third convolution block and a fourth convolution block which are connected in series in sequence. The convolution block does not have a fully connected layer, and each convolution block includes a convolution layer, a normalization layer, an activation function and a pooling layer.

6. The large model driven multimodal semantic enhancement domain generalization method according to claim 5, characterized in that: The first convolution block includes a convolution layer C1_1, a normalization layer B1_1, an activation function S1_1, a convolution layer C1_2, a normalization layer B1_2, an activation function S1_2 and a pooling layer M1; The second convolution block includes a convolution layer C2_1, a normalization layer B2_1, an activation function S2_1, a convolution layer C2_2, a normalization layer B2_2, an activation function S2_2 and a pooling layer M2; The third convolution block includes a convolution layer C3_1, a normalization layer B3_1, an activation function S3_1, a convolution layer C3_2, a normalization layer B3_2, an activation function S3_2, a convolution layer C3_3, a normalization layer B3_3, an activation function S3_3 and a pooling layer M3; The fourth convolution block includes a convolution layer C4_1, a normalization layer B4_1, an activation function S4_1, a convolution layer C4_2, a normalization layer B4_2, an activation function S4_2, a convolution layer C4_3, a normalization layer B4_3, an activation function S4_3 and a pooling layer M4.

7. The large model driven multimodal semantic enhancement domain generalization method according to claim 6, characterized in that: The convolution kernel size in each convolution block is 3*3, and the step size is 1; The pooling layer in each convolution block uses the maximum pooling method, and the pooling window size is set to 2*2 and the step size is 2.

8. The large model driven multimodal semantic enhancement domain generalization method according to claim 7, characterized in that: The activation function in each convolution block is the SeLU function, and its calculation formula is: Among them, x is the data to be processed, f(x) is the result after processing, and λ and α are two constants.

9. The large model driven multimodal semantic enhancement domain generalization method according to claim 8, characterized in that: The clustering screening process of the image generated by S4 in S5 by combining the KMeans algorithm includes the following steps: Perform feature extraction on the image, that is, extract features that can describe the image content from the image and express them in the form of vectors; The Euclidean distance is used as a metric to evaluate the similarity between vectors. The calculation formula of the Euclidean distance is: Among them, A and B are any two vectors, a i is the i-th coordinate of vector A, b i is the i-th coordinate of vector B.

10. The large model driven multimodal semantic enhanced domain generalization method according to claim 9, characterized in that: The S6 comprises the following steps: The filtered images in S5 are merged with the dataset used as the training domain in S1. The training domain data and filtered images of the same category are put together and used as the training data for the backbone network as a whole.

Images