A Multimodal Data Generation and Fine-Tuning Method for Domain Images

Abstract

The present invention provides a method for generating and fine-tuning multimodal data of domain images, belonging to the field of artificial intelligence technology, including: preliminarily processing domain images using a visual multimodal model to obtain a general description, performing natural language conversion on set tags to obtain a converted text, and semantically integrating and expanding the image description and the converted text based on a large language model to generate an initial description; extracting the domain images in multiple layers according to the description dimension to obtain layer descriptions for each layer, and combining the initial description to generate a comprehensive description; pairing the domain images with the corresponding comprehensive descriptions to obtain multimodal data; using the multimodal data to perform fine-tuning processing on the visual multimodal model to obtain an optimized multimodal model, thereby enhancing its recognition and question-answering capabilities in domain images.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and particularly relates to a method for generating and fine-tuning multimodal data of domain images. Background Art

[0002] Visual multimodal models (such as gpt-4o / qwen-vl, etc.) can combine text and image information to achieve vision-based question-answering functions. Compared with traditional image classification and image recognition methods, such models have a better user experience and stronger image understanding ability. However, the current general visual multimodal models still perform unsatisfactorily on domain-specific images. This is mainly because the professional domain image data involved in the training process of general models is less, resulting in insufficient capabilities in these fields.

[0003] For example, in professional fields such as biological species classification, general visual multimodal models may be difficult to provide sufficiently accurate answers. And in each professional field, although there are a large number of domain image datasets (such as image classification, object detection datasets), these data are usually unimodal, that is, the images only correspond to numerical labels and lack detailed text descriptions, making it difficult to directly use them for the training of visual multimodal models.

[0004] In summary, how to convert unimodal domain image data into image-text multimodal data that can be used for multimodal model training, and fine-tune the model with this data to improve its capabilities in professional fields is an urgent problem to be solved.

[0005] Therefore, the present invention proposes a method for generating and fine-tuning multimodal data of domain images. Summary of the Invention

[0006] The present invention provides a method for generating and fine-tuning multimodal data of domain images, which is used to process domain images by using existing multimodal models and large language models, and combine multi-layer extraction to generate multimodal data combining images and text, and fine-tune the general visual multimodal model based on this, so as to improve its recognition and question-answering capabilities in domain images.

[0007] The present invention provides a method for generating and fine-tuning multimodal data of domain images, including:

[0008] Step 1: Use a visual multimodal model to preliminarily process the domain image to obtain a general description, where the general description includes: an image description and a set label for the image description;

[0009] Step 2: Perform natural language conversion on the set label to obtain a conversion text, and semantically integrate and expand the image description and the conversion text based on a large language model to generate an initial description;

[0010] Step 3: Perform multi-layer extraction on the domain image according to the description dimension to obtain the layer description of each layer, and generate a comprehensive description in combination with the initial description;

[0011] Step 4: Pair the domain image with the corresponding comprehensive description to obtain multi-modal data;

[0012] Step 5: Use the multi-modal data to perform fine-tuning processing on the visual multi-modal model to obtain an optimized multi-modal model.

[0013] Preferably, performing multi-layer extraction on the domain image according to the description dimension to obtain the layer description of each layer includes:

[0014] Determine the domain type of the domain image, and match the description dimension and the description definition based on each description dimension from the type-dimension comparison table;

[0015] Respectively match the dimension extraction model from the type-definition-recognition comparison table according to the description definition, perform layer extraction on the domain image to obtain the extraction layer under the corresponding description dimension, and then perform meaning analysis on the extraction layer to obtain the layer description.

[0016] Preferably, performing meaning analysis on the extraction layer to obtain the layer description includes:

[0017] Retrieve the process log of the layer extraction of the domain image by the dimension extraction model, and respectively determine the extraction key coefficients of each position point in the corresponding extraction layer;

[0018] Divide the extraction key coefficients according to the extraction threshold matched with the corresponding description dimension, and regard the surface composed of the position points greater than or equal to the extraction threshold as the first layer, and regard the surface composed of the position points less than the extraction threshold as the second layer;

[0019] Perform global feature analysis on the corresponding extraction layer to obtain the first feature. At the same time, perform global feature analysis on the corresponding first layer and second layer respectively to obtain the corresponding second feature and third feature;

[0020] Perform natural language processing on the first feature, second feature, and third feature respectively to obtain the corresponding first description, second description, and third description;

[0021] Establish a first difference function between the first description and the second description, a second difference function between the first description and the third description, and a fusion function between the second description and the third description;

[0022] Obtain supplementary features according to the first difference function, second difference function, and fusion function;

[0023] Analyze the meanings of the first feature and the supplementary features to obtain the corresponding layer descriptions.

[0024] Preferably, generate a comprehensive description by combining the initial description, including:

[0025] If there is only 1 description dimension in the domain type, at this time, regard the corresponding layer description as the description to be analyzed;

[0026] If there are multiple description dimensions in the domain type, at this time, perform description fusion on the layer descriptions under all description dimensions to obtain the description to be analyzed;

[0027] Generate a comprehensive description based on the description to be analyzed and in combination with the initial description.

[0028] Preferably, perform description fusion on the layer descriptions under all description dimensions to obtain the description to be analyzed, including:

[0029] Extract the quantity words and category words in each layer description to construct a description vector;

[0030] Based on deep learning technology, mine the interest definitions of the category words under each description dimension, and obtain the interested words under the corresponding category words, and then construct an extended vector;

[0031] Determine the interest types of the interested words under each category word in the extended vector, and count the number of occurrences of the same interest type in the extended vector;

[0032] Select the interest type with the largest statistical count, and match the corresponding attention mechanism from the type-mechanism comparison table;

[0033] Perform interested description embedding on the layer descriptions of all extraction layers based on the attention mechanism under each description layer, and then perform description fusion to obtain the description to be analyzed.

[0034] Preferably, obtain the interested words under the corresponding category words, including:

[0035] Respectively use the category words under the description dimension as indexes, and in combination with deep learning technology, mine the strongly related definitions based on each index under the corresponding description dimension from the specified database;

[0036] Perform clustering analysis on all the strongly related definitions under each index to obtain a definition cluster, and regard the referring definition of the corresponding definition cluster as the interest definition, where the number of the definition clusters is at least 1.

[0037] Preferably, use the multi-modal data to perform fine-tuning processing on the visual multi-modal model, including:

[0038] Convert the multi-modal data into the current input sequence that matches the visual multi-modal model;

[0039] Convert the general description of the domain image into the original input sequence that matches the visual multi-modal model;

[0040] Compare and analyze the original input sequence and the current input sequence of the same domain image to determine the difference sequence and the first number Nd of the difference sequence, and according to Construct a division window according to the obtained second number, where represents the ceiling symbol; represents taking a random positive integer from the range ;

[0041] Lock the start position where the difference sequence first appears and the end position where the difference sequence last appears in the current input sequence to obtain a position sequence segment;

[0042] If the segment window of the position sequence segment is the same size as the division window, keep the current input sequence unchanged;

[0043] If they are not the same, divide the position sequence segment in turn according to the division window at the start position to obtain several combined sequences, and combine each combined sequence, the sequence before the start position, and the sequence after the end position respectively to obtain a new sequence;

[0044] Perform natural language processing on the new sequence. If the semantics of the new sequence is similar to the semantics of the current input sequence, keep the corresponding new sequence;

[0045] Otherwise, screen the position where the difference sequence appears for the second time, and combine it with the end position where the difference sequence appears for the last time to re-judge the new sequence obtained subsequently;

[0046] Train the visual multi-modal model based on all the retained new sequences and all the current input sequences.

[0047] Preferably, based on a large language model, semantically integrate and expand the image description and the converted text to generate an initial description, including:

[0048] Based on the large language model, find the relevant concepts in the image description and the converted text and establish a corresponding relationship;

[0049] Based on the corresponding relationship, fuse the complementary information in the image description and the converted text;

[0050] Use an external knowledge graph to search for newly introduced concepts and knowledge related to the fused semantics, and combine the background knowledge of the domain image to determine the depth and breadth of the semantics to generate an integrated and expanded text;

[0051] Perform a natural language rephrasing on the integrated and extended text to obtain an initial description.

[0052] Compared with the prior art, the beneficial effects of the present application are as follows:

[0053] By using existing multi-modal models and large language models to process domain images, and combining multi-layer extraction, multi-modal data combining images and text is generated, and based on this, a general vision multi-modal model is fine-tuned, thereby improving its recognition and question-answering capabilities in domain images.

[0054] Other features and advantages of the present invention will be described in the following specification, and in part, will be obvious from the specification, or will be understood by implementing the present invention. The objectives and other advantages of the present invention can be achieved and obtained by the structures specifically pointed out in the written specification and the drawings.

[0055] The technical solutions of the present invention will be further described in detail below through the drawings and embodiments. Description of the Drawings

[0056] The drawings are used to provide a further understanding of the present invention, and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention, and do not constitute a limitation to the present invention. In the drawings:

[0057] Figure 1 is a flowchart of a method for generating and fine-tuning multi-modal data of a domain image in an embodiment of the present invention. Detailed Embodiments

[0058] The following describes the preferred embodiments of the present invention with reference to the drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0059] The present invention provides a method for generating and fine-tuning multi-modal data of a domain image, as Figure 1 shown, including:

[0060] Step 1: Use a vision multi-modal model to perform preliminary processing on a domain image to obtain a general description, where the general description includes: an image description and a set label for the image description;

[0061] Step 2: Perform natural language conversion on the set label to obtain a conversion text, and based on a large language model, perform semantic integration and extension on the image description and the conversion text to generate an initial description;

[0062] Step 3: Perform multi-layer extraction on the domain image according to the description dimension to obtain a layer description for each layer, and combine the initial description to generate a comprehensive description;

[0063] Step 4: Pair the domain image with the corresponding comprehensive description to obtain multimodal data;

[0064] Step 5: Use the multimodal data to fine-tune the visual multimodal model to obtain an optimized multimodal model.

[0065] In this embodiment, the visual multimodal model can be the Qwen2.5-VL-7B-Instruc model. To obtain labeled image data in the domain, for example, image classification data of fungi.

[0066] Send the picture and the prompt "Please describe this picture" to the visual multimodal model to obtain an image description, such as "There is a mushroom in this picture."

[0067] In this embodiment, the set labels of the image description can be numerical or character labels.

[0068] In this embodiment, natural language conversion is performed on the set labels to obtain conversion text. For example, in an image classification dataset, the labels of images are usually numbers (e.g., label "1" represents a cat, "2" represents a dog, etc.). By designing natural language templates, these numerical labels are converted into meaningful text descriptions. For example, label "1" can be converted to "The species in this image is a cat", and label "2" can be converted to "This is a dog". This conversion helps to turn simple numerical labels into understandable text information, facilitating combination with image descriptions.

[0069] In this embodiment, the tuning framework for the model generally includes: a Vision Encoder, a Large Language Model (LLM), and an Adapter.

[0070] The Vision Encoder is used to extract features from images. The Adapter serves as a bridge to convert image features into the word embedding space, thereby facilitating the LLM's interpretation of the output of the Vision Encoder. The Adapter is usually designed as a lightweight neural network structure, such as a fully connected network, to ensure efficient multimodal integration. Subsequently, the LLM processes the data combined with text and image features to generate the expected text response.

[0071] Loss function for model training:

[0072] The goal of training is to minimize the gap between the generated text and the true text. Usually, the Cross-Entropy Loss is used to calculate the difference between the probability of the generated text and the true label. Specifically, the model is trained by maximizing the probability of the correct text. The loss function Lgeneration can be expressed as:

[0073] ;

[0074] Where N is the number of samples in a training batch, i represents the i-th sample in the training batch, T is the length of the valid output sequence y of sample i, xi is the feature vector of the image in training batch sample i, and INSTi is the text instruction corresponding to the image in sample i. For example, "Please describe this image". i is the true label of the output sequence of sample i at time step t. is the generated sequence of sample i before time step t. represents that the output sequence time step t of sample i is the true label of the probability. In the training process, the model continuously adjusts the parameters through backpropagation to minimize the loss function.

[0075] At each iteration, the model generates an output response based on the input image and instruction text, then compares it with the true annotation text, calculates the loss, and updates the model parameters.

[0076] After training, the model is evaluated on the validation set. The evaluation metrics include:

[0077] Custom metric: Evaluate the accuracy of the model's prediction of image labels. For example, when inputting a picture of a certain type of mushroom, output a prediction of whether it is edible.

[0078] Since the label information of the image is fused into the image description during the data generation stage, the accuracy of label prediction should be evaluated during prediction. The method can be designed according to specific labels. For example, keyword matching can be used to determine whether the prediction is accurate.

[0079] Common evaluation metrics for text generation tasks such as BLEU or ROUGE.

[0080] The evaluation process includes: obtaining the sample data of the validation set, inputting each sample data (image and text instruction) into the trained multi-modal model to obtain the predicted text output, and then evaluating using custom metrics and common metrics based on the predicted output and the correct output given in the sample.

[0081] In this embodiment, the generated domain multi-modal data is used to fine-tune the general vision multi-modal model through supervised learning.

[0082] Optimize the parameters of the model to make it more suitable for domain image data, thereby improving its image understanding and question-answering capabilities in a specific domain.

[0083]

[0084] ​In this embodiment, it is necessary to analyze the domain image in three dimensions. For example, the environmental scene dimension of the image (such as grassland, sky), the animal scene dimension existing in the image (such as rabbit), and the human scene dimension existing in the image (such as human). Then, based on these three dimensions, layer extraction is performed on the image to obtain a separate image under the corresponding dimension. For example, the extraction layer obtained for the environmental scene dimension only contains grassland, sky, etc. (except rabbits and humans), and so on, and different extraction layers can be obtained.

[0085] In this embodiment, after obtaining the extraction layer, analyzing the image of the corresponding layer can obtain the layer description.

[0086] In this embodiment, the comprehensive description is obtained through the comprehensive processing of all layer descriptions and the initial description.

[0087] In this embodiment, pairing refers to correlating the comprehensive description with the domain image. Multimodal data refers to the comprehensive description of the domain image.

[0088] In this embodiment, since the description richness of multimodal data is higher than the original general description, that is, the sample is optimized. Therefore, by using the multimodal data to continue training the model, the fine-tuning of the model accuracy can be achieved, and then an optimized multimodal model can be obtained.

[0089] The beneficial effects of the above technical solutions are: By using the existing multimodal model and large language model to process the domain image, and combining multi-layer extraction, multimodal data combining images and texts is generated, and based on this, the general vision multimodal model is fine-tuned, so as to improve its recognition and question-answering capabilities in the domain image.

[0090] The present invention provides a method for generating and fine-tuning multimodal data of a domain image. The domain image is subjected to multi-layer extraction according to the description dimension to obtain the layer description of each layer, including:

[0091] Determine the domain type of the domain image, and match the description dimension and the description definition based on each description dimension from the type-dimension comparison table;

[0092] Respectively match the dimension extraction model from the type-definition-recognition comparison table according to the description definition, and perform layer extraction on the domain image to obtain the extraction layer under the corresponding description dimension, and then perform meaning analysis on the extraction layer to obtain the layer description.

[0093] In this embodiment, the domain type refers to outdoor scene type, indoor scene type, bungee jumping scene type, etc., and different types are involved for different scenes.

[0094] In this embodiment, the type-dimension comparison table contains different domain types, as well as set description dimensions and description definitions based on these types, which are stored in advance and can be directly matched.

[0095] In this embodiment, the type-definition-recognition comparison table contains dimension extraction models based on different description definitions under different domain types, which are stored in advance and can be directly matched. Moreover, the dimension extraction models are trained for the neural network model with the original images under the corresponding description dimensions and the scene extraction results under these dimensions as samples.

[0096] The specific content included in the type-definition-recognition comparison table is as follows:

[0097]

[0098] Suppose the input image is a photo of a forest:

[0099] Determine that the domain type is an outdoor scene;

[0100] Search for the description dimension and the model. According to the type-definition-recognition comparison table, find the following information:

[0101] Description dimension 1: Color of the sky

[0102] Description definition: Extract the main color of the sky area

[0103] Dimension extraction model: Color segmentation model (based on CNN)

[0104] Description dimension 2: Texture of the ground

[0105] Description definition: Identify the texture features of the ground area

[0106] Dimension extraction model: Texture classification model (based on ResNet)

[0107] Description dimension 3: Types of plants

[0108] Description definition: Identify the main plant types in the plant area

[0109] Dimension extraction model: Plant classification model (based on YOLOv5)

[0110] Execute the extraction:

[0111] Process the image using the corresponding model:

[0112] Color of the sky: The model outputs "blue".

[0113] Texture of the ground: The model outputs "grassland".

[0114] Types of plants: The model outputs "pine trees".

[0115] Generation layer description:

[0116] Fuse the extraction results to obtain the final layer description:

[0117] "This is a sunny forest scene with a blue sky, the ground covered with grass, and the main plants being pine trees."

[0118] In this embodiment, after using the model to perform scene extraction in the corresponding description dimension, the extraction layer (the scene extraction result based on the model) can be obtained.

[0119] In this embodiment, semantic analysis is performed by dividing the position points by a threshold and then performing feature and description fusion.

[0120] The beneficial effects of the above technical solution are: sequentially obtain the description definition and the related dimension extraction model from two comparison tables, thereby obtaining the extraction layer, and obtaining the layer description through semantic analysis, providing a basis for subsequent obtaining of the comprehensive description.

[0121] The present invention provides a method for generating and fine-tuning multi-modal data of domain images. Performing semantic analysis on the extraction layer to obtain the layer description includes:

[0122] Retrieve the process log of the layer extraction of the domain image by the dimension extraction model, and respectively determine the extraction focus coefficients of each position point in the corresponding extraction layer;

[0123] Divide the extraction focus coefficients according to the extraction threshold matching the corresponding description dimension, and regard the surface composed of the position points greater than or equal to the extraction threshold as the first layer, and regard the surface composed of the position points less than the extraction threshold as the second layer;

[0124] Perform global feature analysis on the corresponding extraction layer to obtain the first feature. At the same time, perform global feature analysis on the corresponding first layer and second layer respectively to obtain the corresponding second feature and third feature;

[0125] Perform natural language processing on the first feature, second feature, and third feature respectively to obtain the corresponding first description, second description, and third description;

[0126] Establish a first difference function between the first description and the second description, a second difference function between the first description and the third description, and a fusion function between the second description and the third description;

[0127] Obtain supplementary features according to the first difference function, second difference function, and fusion function;

[0128] Perform feature semantic analysis on the first feature and the supplementary features to obtain the corresponding layer description.

[0129] In this embodiment, the process log refers to the relevant data generated during the process of the model extracting layers from an image, which can be directly captured. Specifically:

[0130] Before the model starts processing the image, use the time function in the programming language to record the start time. For example, in Python, the time.time( ) function can be used to obtain the current timestamp as the start time start_time.

[0131] At the start of each pixel extraction operation, record the start time pixel_start_time of the pixel extraction. Similarly, the time.time( ) function can be used.

[0132] When the pixel extraction operation ends, record the end time pixel_end_time. Calculate the extraction time of the pixel by pixel_end_time - pixel_start_time and record it in the log.

[0133] Set a counter for each pixel, with an initial value of 0. A two-dimensional array extraction_count with the same size as the image pixel matrix can be used to record the extraction times of each pixel. Each element of the array corresponds to a pixel in the image.

[0134] Whenever an extraction operation is performed on a pixel, increment the corresponding counter value by 1. For example, when extracting the pixel at coordinates (x, y), execute extraction_count[x][y]+=1.

[0135] Create a log file. The open( ) function in Python can be used to open a file in write mode. For example, log_file = open('process_log.txt', 'w').

[0136] During the process of the model extracting layers from the image, when the extraction of a pixel is completed and its extraction time and updated extraction count are obtained, write this information to the log file in a certain format. For example, write it in the format of pixel coordinates, extraction time, extraction count. For example, (10, 20), 0.001, 3 means that the extraction time of the pixel at coordinates (10, 20) is 0.001 seconds and the extraction count is 3 times.

[0137] After the model finishes processing the entire image, close the log file to ensure that all log information is correctly saved. Use log_file.close( ) to close the file.

[0138] The process log contains the extraction time and the number of extractions for each pixel point in the image to determine the extraction focus coefficient for each position point in the corresponding extraction layer. Different image feature extraction algorithms process pixel points differently. For example, the SIFT (Scale-Invariant Feature Transform) algorithm needs to detect stable feature points in the image and calculate their local features, which involves a relatively large amount of computation for each pixel point, a relatively long extraction time, and may sample and calculate certain key pixel points multiple times to ensure the accuracy of the features.

[0139] Among them, the extraction focus coefficient = , where St and Rc respectively represent the extraction time length and the number of extractions for the corresponding position point; T0 and R0 respectively represent the preset time length and the preset number of times;

[0140] In this embodiment, the value of the extraction threshold is 1.

[0141] In this embodiment, the first layer is the surface composed of position points that satisfy being greater than or equal to the extraction threshold;

[0142] The second layer is the surface composed of position points that satisfy being less than the extraction threshold.

[0143] In this embodiment, the first feature, the second feature, and the third feature are obtained based on a feature analysis model (a commonly used convolutional neural network (CNN) model in the field of computer vision. It extracts different levels and types of features from the original image data through operations such as multi-layer convolution and pooling on the image. For example, the output of the fully connected layer of a convolutional neural network (CNN) is a feature vector. For instance, after a trained ResNet model extracts features from an image of a cat, it outputs a 1000-dimensional vector, and each value in the vector represents the response intensity of the image in the corresponding feature dimension, which may include numerical representations of relevant features such as the shape of the cat's ears, the color of its eyes, and the texture of its fur), which is prior art. Image features refer to a set of attributes that can characterize the characteristics or content of an image, mainly including image natural features (such as brightness, color, texture, etc.), and then relevant descriptions can be directly obtained through natural language processing. Specifically, natural language processing technology analyzes the identified image features. It is based on a large amount of training data and complex algorithm models and can convert these feature information into natural language. For example, it will generate a simple and clear natural language description such as "There is a mushroom in the picture" according to the natural features such as the shape and color of the mushroom in the image.

[0144] In this embodiment, the first difference function = G (the first description, the second description), the second difference function = G (the first description, the third description), and the fusion function = R (the second description, the third description). It should be noted that the fusion function is obtained by processing based on a relevant fusion model, and the fusion model is trained for a neural network model with different descriptions and the results of description combinations as samples.

[0145] The difference function is obtained by analyzing based on a difference model, and the difference model is trained for a neural network model with different descriptions and description differences as samples.

[0146] For example, the first description is: description u1, description u2, description u3, description u4, and the second description is description u1, description u2. At this time, the result obtained by the first difference function is: description u3, description u4.

[0147] In this embodiment, the supplementary feature is regarded as the image feature under the inconsistent description corresponding to the difference function and the fusion function. Specifically, the inconsistent description is mapped to the image for comparison, and then the feature of the image information under the inconsistent description is obtained.

[0148] In this embodiment, the first feature and the supplementary feature are input into the meaning analysis model to obtain a layer description, and the meaning analysis model is trained for a neural network model with different feature combinations and the analysis results based on these combinations as samples. For example, the layer description for the environmental scene is: The environmental beautification makes people feel relaxed and happy.

[0149] The beneficial effects of the above technical solution are: Based on the process log, the extraction key coefficient of each position point is determined, and then the coefficient division is realized through the extraction threshold, effectively determining the features of different layers. And through the analysis of the three-layer features and natural language processing, a layer description is obtained, ensuring the accuracy of description acquisition.

[0150] The present invention provides a method for generating and fine-tuning multi-modal data of domain images, which combines the initial description to generate a comprehensive description, including:

[0151] If there is only 1 description dimension under the domain type, at this time, the corresponding layer description is regarded as the description to be analyzed;

[0152] If there are multiple description dimensions under the domain type, at this time, the layer descriptions under all description dimensions are fused to obtain the description to be analyzed;

[0153] Based on the description to be analyzed and combined with the initial description, a comprehensive description is generated.

[0154] In this embodiment, the comprehensive description refers to obtaining the comprehensive description through the description union processing of the description to be analyzed and the initial description.

[0155] The beneficial effects of the above technical solution are as follows: By performing quantitative analysis on dimensions, the rationality of obtaining the description to be analyzed is realized, and then a comprehensive description is obtained.

[0156] The present invention provides a method for generating and fine-tuning multi-modal data of domain images, which performs description fusion on layer descriptions under all description dimensions to obtain a description to be analyzed, including:

[0157] Extract the quantitative words and category words in each layer description to construct a description vector;

[0158] Based on deep learning technology, mine the interest definition of category words under each description dimension, and obtain the interesting words under the corresponding category words, and then construct an extended vector;

[0159] Determine the interesting type of each category word in the extended vector for the interesting words, and count the number of occurrences of the same interesting type in the extended vector;

[0160] Select the interesting type with the largest statistical count, and match the corresponding attention mechanism from the type-mechanism comparison table;

[0161] Based on the attention mechanism under each description layer, perform interesting description embedding on the layer descriptions of all extraction layers, and then perform description fusion to obtain a description to be analyzed.

[0162] In this embodiment, the description vector = {category words in the layer description, quantitative words under the corresponding category words}.

[0163] In this embodiment, the extended vector = {interesting words under the category words}, and the extended vector includes the words involved in the corresponding layer description.

[0164] In this embodiment, under category word 1: interesting word 11, interesting word 12;

[0165] Under category word 2: interesting word 21, sensing area word 22, interesting word 23;

[0166] Interesting type A1 includes: interesting word 11, interesting word 21, sensing area word 22;

[0167] Interesting type A2 includes: interesting word 12, interesting word 23;

[0168] At this time, the number of words under the statistically obtained interesting type A1 is 3, and the number of words under the interesting type A2 is 2.

[0169] In this embodiment, the type-mechanism correspondence table includes different types of interest and the attention mechanism for that type, and the attention mechanism is determined in advance for that type.

[0170] In this embodiment, the embedding of the interest description means that since the attention mechanism allows the model to focus on the key parts of the input text, by calculating the attention weights at each position, it is determined which information is more important for generating the interest description. Then, based on these weights, the features of the input text are integrated, and the information related to the content of interest is effectively embedded into a low-dimensional vector space to form the embedded representation of the interest description.

[0171] In this embodiment, the description fusion is implemented based on the fusion description model, and the fusion description model is obtained by training a neural network model with the embedded interest descriptions of the description layers under different description dimensions and the related fusion results as samples.

[0172] The beneficial effects of the above technical solution are: by constructing the description vector and mining the interest definition, an augmented vector is constructed, which provides a basis for determining the largest type of interest subsequently, and the attention mechanism is obtained to realize the reasonable acquisition of the embedded interest description, ensuring the diversity and integrity of the information in the description to be analyzed.

[0173] The present invention provides a method for generating and fine-tuning multi-modal data of domain images to obtain interest words under corresponding category words, including:

[0174] Using the category words under the description dimension as indexes respectively, and combining deep learning techniques to mine the strongly related definitions based on each index under the corresponding description dimension from a specified database;

[0175] Performing cluster analysis on all the strongly related definitions under each index to obtain definition clusters, and regarding the representative definitions of the corresponding definition clusters as interest definitions, where the number of the definition clusters is at least 1.

[0176] In this embodiment, the specified database is pre-stored with relevant domain text information including interest words under different categories, and then the strongly related definitions can be obtained by mining the text information.

[0177] In this embodiment, the strongly related definition means that the correlation coefficient between the words in the specified database and the corresponding index is greater than 0.9, and the calculation method of the correlation coefficient is existing.

[0178] In this embodiment, if there are 10 strongly related definitions under an index, then cluster analysis is performed on these 10 strongly related definitions to obtain the existing clusters, which are regarded as definition clusters.

[0179] The beneficial effects of the above technical solution are as follows: Using category vocabulary as an index and combining deep learning technology to mine strongly related definitions, and then through clustering analysis, the interest definitions are effectively obtained.

[0180] The present invention provides a method for generating and fine-tuning multimodal data of domain images, and using the multimodal data to perform fine-tuning processing on the visual multimodal model, including:

[0181] Converting the multimodal data into a current input sequence matching the visual multimodal model;

[0182] Converting the general description of the domain image into an original input sequence matching the visual multimodal model;

[0183] Comparing and analyzing the original input sequence and the current input sequence of the same domain image to determine the difference sequence and the first number Nd of the difference sequence, and according to Constructing a division window according to the obtained second number, where, represents the ceiling symbol; represents randomly extracting 1 positive integer from the range ;

[0184] Locking the first position where the difference sequence first appears and the last position where the difference sequence last appears in the current input sequence to obtain a position sequence segment;

[0185] If the segment window of the position sequence segment is the same as the size of the division window, keep the current input sequence unchanged;

[0186] If not, divide the position sequence segment in turn according to the division window at the first position to obtain several combined sequences, and respectively combine each combined sequence, the sequence before the first position, and the sequence after the last position to obtain a new sequence;

[0187] Performing natural language processing on the new sequence. If the semantics of the new sequence is similar to the semantics of the current input sequence, keep the corresponding new sequence;

[0188] Otherwise, screen the position where the difference sequence appears for the second time, and in combination with the last position where the difference sequence appears, re-judge the newly obtained sequence;

[0189] Based on all the retained new sequences and all the current input sequences, training the visual multimodal model.

[0190] In this embodiment, the multimodal data conversion is to ensure that the input format meets the input format of the visual multimodal model, so as to obtain the current input sequence, and the general description is similar.

[0191] In this embodiment, for example, when the current input sequence = {r1 r2 r3 r4 r5 r6} and the original input sequence = {r1 r6 r7}, at this time, the obtained difference sequence = {r1 - r1 r2 r3 r4 r5 r6 - r6 r7}. At this time, the first number Nd obtained is 5. At this time, the first position where the difference sequence appears for the first time is the position corresponding to r2, and the last position where the difference sequence appears for the last time is the position corresponding to r7. The position sequence segment is {r2 r3 r4 r5 r6 - r6 r7}. At this time, if the size of the partitioning window is 2, the obtained combined sequences are {r2 r3}, {r4 r5}, {r6 - r6 r7} in sequence, and the obtained new sequences are: {r1 r2 r3}, {r1, r4 r5}, {r1, r6, r7} in sequence.

[0192] In this embodiment, semantic similarity means that after natural language processing, it is found that the semantic similarity between the new sequence and the semantic of the current input sequence is greater than 80% or more, and at this time, it is regarded as similar.

[0193] The beneficial effects of the above technical solution are: by obtaining multi-modal data and the input sequence of general description, and then determining the position sequence segment through sequence difference comparison analysis, and combining the size of the partitioning window to obtain a new sequence, providing more samples for the training of the visual multi-modal model, and ensuring the reliability of the samples through semantic similarity analysis, and then realizing the fine-tuning process of the model.

[0194] The present invention provides a method for generating and fine-tuning multi-modal data of domain images, which integrates and expands the semantics of image descriptions and conversion texts based on a large language model to generate an initial description, including:

[0195] Finding relevant concepts in the image description and conversion text based on the large language model and establishing a corresponding relationship;

[0196] Based on the corresponding relationship, fusing the complementary information in the image description and conversion text;

[0197] Using an external knowledge graph to find newly introduced concepts and knowledge related to the fused semantics, and combining the background knowledge of the domain image to determine the depth and breadth of the semantics, and generating an integrated and expanded text;

[0198] Reformulating the integrated and expanded text in natural language to obtain an initial description.

[0199] In this embodiment, the relevant concepts are all pre-set based on the large prediction model. For example, the image description is: a scene where a little dog is lying in the grass, and the conversion text is: a little dog is lying in the grass. At this time, the image description and the corresponding conversion text are the corresponding relationship.

[0200] In this embodiment, semantic fusion combines multiple knowledge graphs, unifying entities, concepts, and relationships in different knowledge graphs. The entity alignment problem needs to be solved, that is, identifying nodes representing the same real-world entity in different knowledge graphs and merging or establishing associations between them to form a more comprehensive and integrated knowledge graph, thereby obtaining fusion information.

[0201] In this embodiment, newly introduced concepts and knowledge refer to supplementary new concepts and new information. For example, when generating integrated and extended text, the newly introduced concepts and knowledge found are incorporated into it, such as "Currently, for the treatment of diabetes, in addition to traditional drugs, the newly launched [drug name] has shown good efficacy, and its mechanism of action is [specific mechanism]. In terms of combination therapy, the combination plan of [drug combination] can more effectively control blood sugar levels, but attention should be paid to [precautions]." At the same time, the deepening of semantics by combining image background knowledge is also reflected in the text, such as "When there is a pulmonary shadow in the image, if the shadow is located in [specific lung lobe position] and presents [describe the characteristics of the shadow shape, boundary, etc.], it may be [specific disease], and subsequent [other examination items] need to be combined for further diagnosis." In this way, through the collaborative use of external knowledge graphs and domain image background knowledge, in-depth semantic mining and rich text expansion are achieved.

[0202] In this embodiment, rephrasing is to make the description of the integrated and extended text more fluent.

[0203] The beneficial effects of the above technical solutions are as follows: By establishing the correspondence between descriptions and texts through large language models, and then subsequent information fusion, graph searching, background knowledge assistance, etc., integrated and extended texts are obtained to determine the information integrity of the texts. Moreover, through the re-expression in natural language, the description becomes more fluent, providing reliable samples for subsequent model training.

[0204] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalent technologies, the present invention also intends to include these changes and modifications.

Claims

1. A multimodal data generation and fine-tuning method for domain images, characterized in that Including: Step 1: Use a visual multi-modal model to preliminarily process the domain image to obtain a general description, where the general description includes: an image description and a set label for the image description; Step 2: Perform natural language conversion on the set label to obtain a conversion text, and semantically integrate and expand the image description and the conversion text based on a large language model to generate an initial description; Step 3: Perform multi-layer extraction on the domain image according to the description dimension to obtain a layer description for each layer, and generate a comprehensive description in combination with the initial description; Step 4: Pair the domain image with the corresponding comprehensive description to obtain multi-modal data; Step 5: Use the multi-modal data to fine-tune the visual multi-modal model to obtain an optimized multi-modal model; Among them, performing multi-layer extraction on the domain image according to the description dimension to obtain a layer description for each layer includes: Determine the domain type of the domain image, and match the description dimension and the description definition based on each description dimension from the type-dimension comparison table; Respectively match the dimension extraction model from the type-definition-recognition comparison table according to the description definition, and perform layer extraction on the domain image to obtain an extraction layer under the corresponding description dimension, and then perform meaning analysis on the extraction layer to obtain a layer description.

2. The multimodal data generation and fine-tuning method for field images according to claim 1, wherein Performing meaning analysis on the extraction layer to obtain a layer description includes: Retrieve the process log of the layer extraction of the domain image by the dimension extraction model, and respectively determine the extraction focus coefficient of each position point in the corresponding extraction layer; Divide the extraction focus coefficient according to the extraction threshold matched with the corresponding description dimension, and regard the surface composed of position points greater than or equal to the extraction threshold as the first layer, and regard the surface composed of position points less than the extraction threshold as the second layer; Perform global feature analysis on the corresponding extraction layer to obtain a first feature. At the same time, perform global feature analysis on the corresponding first layer and second layer respectively to obtain corresponding second features and third features; Perform natural language processing on the first feature, the second feature, and the third feature respectively to obtain corresponding first descriptions, second descriptions, and third descriptions; Establish a first difference function between the first description and the second description, a second difference function between the first description and the third description, and a fusion function between the second description and the third description; Obtain a supplementary feature according to the first difference function, the second difference function, and the fusion function; Perform feature meaning analysis on the first feature and the supplementary feature to obtain a corresponding layer description.

3. The multimodal data generation and fine-tuning method for domain images according to claim 1, characterized in that Generating a comprehensive description in combination with the initial description includes: If there is only 1 description dimension under the domain type, at this time, regard the corresponding layer description as the description to be analyzed; If there are multiple description dimensions under the domain type, at this time, perform description fusion on the layer descriptions under all description dimensions to obtain the description to be analyzed; Generate a comprehensive description based on the description to be analyzed and in combination with the initial description.

4. The multimodal data generation and fine-tuning method for field images according to claim 3, characterized in that Performing description fusion on the layer descriptions under all description dimensions to obtain the description to be analyzed includes: Extract the quantity vocabulary and category vocabulary in each layer description to construct a description vector; Mining the interest definition of category words under each description dimension based on deep learning technology, obtaining the interested words under the corresponding category words, and then constructing an extended vector; Determining the interest type of each category word in the extended vector for the interested words, and counting the number of occurrences of the same interest type in the extended vector; Screening the interest type with the largest statistical count and matching the corresponding attention mechanism from the type-mechanism look-up table; Performing interested description embedding on the layer descriptions of all extraction layers based on the attention mechanism under each description layer, and then performing description fusion to obtain the description to be analyzed.

5. The multimodal data generation and fine-tuning method for domain images according to claim 4, wherein Obtaining the interested words under the corresponding category words, including: Using the category words under the description dimension as indexes respectively, and mining the strongly related definitions based on each index under the corresponding description dimension from the specified database in combination with deep learning technology; Performing clustering analysis on all the strongly related definitions under each index to obtain definition clusters, and regarding the reference definitions of the corresponding definition clusters as interest definitions, where the number of the definition clusters is at least 1.

6. The multimodal data generation and fine-tuning method for field images according to claim 1, characterized in that Fine-tuning the visual multi-modal model using the multi-modal data, including: Converting the multi-modal data into the current input sequence matching the visual multi-modal model; Converting the general description of the domain image into the original input sequence matching the visual multi-modal model; Compare and analyze the original input sequence and the current input sequence of images in the same field, determine the difference sequence and the first number Nd of the difference sequence, and according to Construct a division window based on the obtained second number, where represents the ceiling symbol; represents randomly selecting 1 positive integer from the range ; Locking the start position of the first occurrence of the difference sequence and the end position of the last occurrence of the difference sequence in the current input sequence to obtain a position sequence segment; If the segment window of the position sequence segment is the same as the size of the division window, keeping the current input sequence unchanged; If they are inconsistent, sequentially dividing the position sequence segment according to the division window at the start position to obtain several combined sequences, and respectively combining each combined sequence, the sequence before the start position, and the sequence after the end position to obtain a new sequence; Performing natural language processing on the new sequence. If the semantics of the new sequence are similar to those of the current input sequence, keeping the corresponding new sequence; Otherwise, screening the position of the second occurrence of the difference sequence, and re-judging the subsequent obtained new sequences in combination with the end position of the last occurrence of the difference sequence; Training the visual multi-modal model based on all the retained new sequences and all the current input sequences.

7. The multimodal data generation and fine-tuning method for field images according to claim 6, wherein Semantically integrating and expanding the image description and the converted text based on the large language model to generate an initial description, including: Searching for relevant concepts in the image description and the converted text based on the large language model and establishing a corresponding relationship; Fusing the complementary information in the image description and the converted text based on the corresponding relationship; Using the external knowledge graph to search for newly introduced concepts and knowledge related to the fused semantics, and determining the depth and breadth of the semantics in combination with the background knowledge of the domain image to generate an integrated and expanded text; Performing a natural language re-expression on the integrated and expanded text to obtain an initial description.

Images