Multi-modal human body large model training method and system considering multi-granularity semantic alignment

Abstract

The invention relates to the technical field of artificial intelligence multi-modal large models, and discloses a multi-modal human body large model training method and system considering multi-granularity semantic alignment, and the method comprises the steps: carrying out the prefix construction of a large-scale training set through improving a network structure, adding prefixes related to description to the data in the training set based on human body features; and training the multi-modal human body large model in batches, pairing each image and each text in each batch of training data, improving comparison loss through similarity calculation and semantic relaxation, and optimizing visual and text modal encoder parameters, so as to improve the accuracy of the multi-modal human body large model. According to the multi-modal human body large model training system considering multi-granularity semantic alignment, aiming at the multi-granularity semantic alignment problem encountered by a current multi-modal large model in human body cross-modal retrieval application, a general multi-modal large model is improved from two aspects of a network structure and a loss function; and the performance of cross-modal human body retrieval under multi-granularity semantics is enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of artificial intelligence multimodal large models, and in particular to a multimodal human body large model training method and system considering multi-granularity semantic alignment. Background Art

[0002] With the rapid development of artificial intelligence technology, multimodal large models have attracted much attention due to their powerful ability to process multiple data types such as images, text, audio, etc. Especially in the field of cross-modal retrieval of human bodies, multimodal large models have broad application prospects and can realize retrieval tasks from natural language feature descriptions to human body images.

[0003] At present, the network structure of mainstream multimodal large models (such as BLIP and QWEN) can be abstracted into three modules, including pre-trained modal encoders, pre-trained large language models, and modal adaptation interfaces connecting them. Figure 1 As shown in the figure, the modal encoder compresses the original information (such as images or audio) into a more compact representation. A common method is to use a pre-trained encoder (such as CLIP) that has been aligned with other modalities to reduce the difficulty of later training alignment with a large language model. The pre-trained large language model has been embedded with rich world knowledge through pre-training on a massive network corpus, and has emerged with powerful generalization and reasoning capabilities. The large language model can only perceive text, so only by aligning other modal encodings can the capabilities of the large language model be brought into play. However, training a multimodal model in an end-to-end manner is extremely expensive and difficult to train. The mainstream method is to introduce a learnable connector between the pre-trained visual encoder and the large language model to efficiently project the information into the embedding space that the large language model can understand to complete the alignment between the modalities.

[0004] In terms of training strategy, multimodal large model training can be roughly divided into the comparison pre-training stage, instruction fine-tuning stage and alignment fine-tuning stage. Figure 2 As shown. In the pre-training stage, a large number of image-text pairs are used to align the image information to the representation space of the large language model, so that the large language model can initially understand the visual token. This stage can usually be divided into two sub-stages: comparative pre-training and generative pre-training. Through this progressive training method, the convergence of the multimodal large model training process is more stable; in the instruction fine-tuning stage, the performance of the model on downstream tasks is improved through a variety of task data of various types, so that the model can understand and obey complex instructions; in the alignment fine-tuning stage, reinforcement learning technology is generally used to align the model with human values ​​or certain specific needs (such as fewer hallucinations).

[0005] Thanks to massive data sets and ultra-large-scale parameters, large pre-trained models have shown excellent performance in cross-modal retrieval tasks. Compared with general retrieval tasks, cross-modal retrieval for human bodies usually requires multi-granularity semantic alignment capabilities. For example, in actual human body retrieval applications, for the same human body image, the text description usually has semantic descriptions of multiple granularities. Figure 3 -a, "grey top", "grey top and black shorts" and "grey top and black shorts and red shoes" are all correct descriptions that can detect the person shown in the picture. From the contrast characteristics of the cross-modal retrieval task itself, it can be seen that its retrieval accuracy is mainly determined by the contrast pre-training stage. However, the mainstream multimodal large models have the following problems in human retrieval applications: the contrast pre-training stage usually focuses on the coarse-grained alignment at the instance level, while human retrieval pays more attention to the attribute granularity of the instance at the instance level. The linear expression of the contrast loss used for modal feature alignment in the pre-training stage cannot distinguish between multiple semantic granularities, which not only affects the stability of the training but also easily causes the model to overfit to the training data, resulting in serious degradation of the model performance. Such as Figure 3 As shown in the figure, when the text describes "there is a person wearing a gray shirt in the picture", Figure 3 -a with Figure 3 -b are all images with their correct descriptions, and they are mapped to a unified embedding space during the modality alignment process, labeled as t1, t2, v1, v2, respectively. Figure 3 -c (where t1=t2=t), Figure 3 -d shows the update diagram of the image encoder during the back propagation process of contrast training; similarly, for the same image, there are multiple descriptions with different semantics but all of them are true. Figure 4 -a, 4-b, and 4-c depict the update direction of the text encoder (large language model) during the back propagation of contrastive training. Therefore, in the process of human body image-text contrast training, it is easy for each modality encoder to update in the wrong direction. Summary of the invention

[0006] The purpose of the present invention is to solve the shortcomings existing in the above-mentioned prior art, and to provide a multimodal human body large model training method and system considering multi-granularity semantic alignment, respectively making corresponding improvements to the general multimodal large model from two aspects: network structure and loss function, so as to further enhance its performance of cross-modal human body retrieval under multi-granularity semantics.

[0007] A multimodal human body model training method considering multi-granularity semantic alignment is provided, including: Prefix construction is performed on a large-scale training set by improving the network structure, and prefixes describing relevant data in the training set are added based on human body characteristics; The multimodal human body model is trained in batches, each image in each batch of training data is paired with each text, the contrast loss is improved by similarity calculation and semantic relaxation, and the parameters of the visual and text modality encoders are optimized.

[0008] Furthermore, the prefix construction of the large-scale training set by improving the network structure includes: A prefix template learning layer is added between the original embedding layer and the large language model to construct the multimodal human body large model, learn prefix representations in an end-to-end manner, and automatically generate corresponding description-related prefixes for each data.

[0009] Furthermore, the prefix construction of the large-scale training set includes: Preprocessing the images and texts in the training set in the embedding layer, wherein the preprocessing of the images includes cleaning, labeling and normalization, and the preprocessing of the text includes pairing and format conversion, and at the same time, a certain position is reserved at the beginning of the text data for adding a prefix; The prefix template learning layer is based on a neural network structure, receives the text vector preprocessed by the embedding layer, generates a suitable prefix vector after neuron activation and weight adjustment, adds the prefix vector to the reserved prefix position, rewrites the input text, and generates a text vector with the prefix; After iterative training of the training set data, the prefix template learning layer realizes automatic generation of prefixes.

[0010] Furthermore, the adding of description-related prefixes to the data in the training set based on human body features includes: The text modality is prefix transformed, and the original text is rewritten according to the human body features through template combination. A prefix matching the description is added to the original text. The prefix matches the multi-granularity semantics of the image with the text data of each granularity, so that after multiple rounds of iterations, the descriptions of various granularities of the multimodal human body model collapse to the same embedding layer, avoiding the performance collapse of the multimodal human body model caused by the linear characteristics of multi-granularity semantics and contrast loss.

[0011] Furthermore, the similarity calculation includes: Pair each image in the batch with each text to generate an N×N similarity matrix, where N is the number of samples in the current batch, and each sample includes an image and its corresponding text; Calculate the similarity between each image and its corresponding text, and also calculate the similarity between the image and every other text in its batch; Maximize the similarity of correct pairs while minimizing the similarity of incorrect pairs.

[0012] Furthermore, in the contrastive learning of the multimodal human model, the loss includes the image-to-text cross entropy loss and text-to-image cross entropy loss , expressed as: .

[0013] The improvement of the comparison loss by similarity calculation specifically includes: For a batch of training samples of a class, the image-to-text cross entropy loss The calculation formula is as follows: in, Indicates The predicted probability of image text matching.

[0014] Preferably, the Image text matching prediction probability It is calculated by softmax, and the calculation formula is as follows: in, Represents the feature vector With the eigenvector The inner product of The calculation formula is as follows: in, Indicates image feature vector, Indicates A text feature vector.

[0015] Furthermore, the improvement of contrast loss by semantic relaxation specifically includes: A new length field is added to the original training sample, which is used as the exponential decay factor in the contrast loss. In the early stage of model training, the granularity is approximately characterized by the text length, and different degrees of relaxation are performed for different granularity semantics in the original contrast loss function. Exponential decay is introduced into the calculation formula, and the calculation formula is expressed as follows: in, represents the actual input length of the current text. The hyperparameter T is used to control the attenuation strength. The prefix template learning layer is randomly initialized and fine-tuned. When T is set to infinity, .

[0016] On the other hand, a multimodal human body model training system considering multi-granularity semantic alignment is provided, based on the above method, including: A prefix construction module is used to construct prefixes for a large-scale training set by improving the network structure, and to add description-related prefixes to the data in the training set based on human body features; The contrast loss improvement module is used to train the multimodal human body model in batches, pair each image with each text in each batch of training data, improve the contrast loss through similarity calculation and semantic relaxation, and optimize the visual and text modality encoder parameters.

[0017] Compared with the prior art, the present invention has the following beneficial effects: The present invention improves the network structure and introduces a learnable prefix template learning layer, thereby enhancing the adaptability of the model to the input data, enabling the model to better capture the template features of the input data and perform prefix construction on large-scale training sets, thereby solving the multi-granularity semantic alignment problem encountered by current large multi-modal models in human cross-modal retrieval applications. The present invention improves the contrast loss function through similarity calculation and semantic relaxation, optimizes the image and text modality encoder parameters, and further enhances its performance of cross-modal human body retrieval under multi-granularity semantics; The present invention alleviates the inadaptability of the original base large model in processing fine-grained text by introducing an attenuation factor of the input granularity into the loss function, thereby improving the stability and overall performance of model training, effectively processing multi-granularity input data, and improving the model's ability to fuse information of different granularities. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings: Figure 1 A schematic diagram of a multi-modal large model architecture in the prior art of the present invention; Figure 2 This is a multi-modal large model training process in the prior art of the present invention; Figure 3 It is a schematic diagram of updating a visual modality encoder in the prior art of the present invention; Figure 4 The figure is a schematic diagram of updating a text modal encoder in the prior art of the present invention.

[0019] Figure 5 A flow chart of a multimodal human body model training method considering multi-granularity semantic alignment according to the present invention; Figure 6 A schematic diagram showing a comparison between a prefix structure before and after the present invention; Figure 7 A schematic diagram of a model collapse of the present invention; Figure 8 This is a schematic diagram of a model after adding a prefix template learning layer in the present invention; Fig. 9 A schematic diagram of multimodal contrast loss of a human body according to the present invention; Fig.10 This is a schematic diagram of three-stage training of a multimodal human model according to the present invention. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0021] To address the problem of multi-granular semantic alignment encountered by current large multimodal models in human cross-modal retrieval applications, a loss attenuation and input learnable template method for human images and multi-granular semantics was proposed, which effectively alleviated the problem of inconsistency between its training objectives and actual applications.

[0022] The specific implementation of the present invention is described below with reference to the accompanying drawings and embodiments.

[0023] like Figure 5 As shown, this embodiment provides a multimodal human body model training method considering multi-granularity semantic alignment, and the technical solution includes: Prefix construction is performed on a large-scale training set by improving the network structure, and prefixes describing relevant data in the training set are added based on human body characteristics; The multimodal human body model is trained in batches, each image in each batch of training data is paired with each text, the contrast loss is improved by similarity calculation and semantic relaxation, and the parameters of the visual and text modality encoders are optimized.

[0024] The prefix construction of the large-scale training set by improving the network structure includes: A prefix template learning layer is added between the original embedding layer and the large language model, such as Figure 8 As shown, the multimodal human body model is constructed, and prefix representation is learned in an end-to-end manner to automatically generate corresponding description-related prefixes for each data.

[0025] At the same time, in this embodiment, a new attention layer is added, which can effectively process multi-granularity input data and improve the model's ability to integrate information of different granularities.

[0026] Specifically, the prefix construction of the large-scale training set further includes: Preprocessing the images and texts in the training set in the embedding layer, wherein the preprocessing of the images includes cleaning, labeling and normalization, and the preprocessing of the text includes pairing and format conversion, and at the same time, a certain position is reserved at the beginning of the text data for adding a prefix; The prefix template learning layer is based on a neural network structure, receives the text vector preprocessed by the embedding layer, generates a suitable prefix vector after neuron activation and weight adjustment, adds it to the reserved prefix position, rewrites the input text, and generates a text vector with a prefix; After iterative training of the training set data, the prefix template learning layer realizes automatic generation of prefixes.

[0027] In addition, the adding of description-related prefixes to the data in the training set based on human body features further includes: The text modality is prefix transformed, and the original text is rewritten according to the human body features through template combination. A prefix matching the description is added to the original text. The prefix matches the multi-granularity semantics of the image with the text data of each granularity, so that after multiple rounds of iterations, the descriptions of various granularities of the multimodal human body model collapse to the same embedding layer, avoiding the performance collapse of the multimodal human body model caused by the linear characteristics of multi-granularity semantics and contrast loss.

[0028] In this embodiment, by adding a specific prefix before the input, the "many-to-one" problem existing in multi-granular semantics is transformed into a semantically precise "one-to-one" alignment, thereby solving the performance degradation problem of multimodal large models in human retrieval applications. In the specific task of human image-text modality alignment, compared with text input, the semantics of human images are definite and contain semantic information of all granularities, so only the text modality input is prefixed. For human features {face, head, top, bottom, shoes, accessories, behavior}, the original training data can be transformed into a prefix that matches the description through template combination, such as Figure 6 shown.

[0029] Through this method, the image not only adapts to text data of various granularities, but also avoids the linear characteristics of multi-granularity semantics and contrast loss that cause the model performance to collapse, such as Figure 7As shown in , after multiple iterations of the model, descriptions of various granularities all collapse to the same embedding. However, explicitly constructing prefix templates for each training data in a large-scale dataset is inefficient and labor-intensive. Figure 8 As shown in the figure, a prefix (template) learning layer is added between the original embedding layer and the large language model to replace the manual process in an end-to-end manner, which not only reduces the training cost, but also can learn a more compact prefix representation while reducing the inference overhead (fixed-length learnable prefix).

[0030] Furthermore, the similarity calculation further includes: Pair each image in the batch with each text to generate an N×N similarity matrix, where N is the number of samples in the current batch, and each sample includes an image and its corresponding text; Calculate the similarity between each image (or text) and its corresponding text (or image), and also calculate the similarity between the image (or text) and every other text (or image) in the batch; Maximize the similarity of correct pairs while minimizing the similarity of incorrect pairs.

[0031] Furthermore, if Fig. 9 As shown, in the contrastive learning of the multimodal human body model, the loss includes the image-to-text cross entropy loss and text-to-image cross entropy loss , expressed as: .

[0032] The improvement of the comparison loss by similarity calculation specifically includes: For a batch of training samples of a class, the image-to-text cross entropy loss The calculation formula is as follows: in, Indicates The predicted probability of image text matching.

[0033] The said Image text matching prediction probability It is calculated by softmax, and the calculation formula is as follows: in, Represents the feature vector With the eigenvector The inner product of The calculation formula is as follows: in, Indicates image feature vector, Indicates A text feature vector.

[0034] The improvement of contrast loss by semantic relaxation specifically includes: In order to smooth the training process, in the early stage of training, different degrees of relaxation are required in the original contrast loss function for different granularity semantics, that is, the finer the granularity, the greater the degree of relaxation.

[0035] A new length field is added to the original training sample, which is used as the exponential decay factor in the contrast loss. In the early stage of model training, the granularity is approximately characterized by the text length, and different degrees of relaxation are performed for different granularity semantics in the original contrast loss function. Exponential decay is introduced into the calculation formula, and the calculation formula is expressed as follows: in, represents the actual input length of the current text. The hyperparameter T is used to control the attenuation strength. The prefix template learning layer is randomly initialized and fine-tuned. When T is set to infinity, .

[0036] In summary, in this embodiment, we train the multimodal human body model in three stages, such as Fig.10 As shown, including: S1: Add a new length field to the original image and text training samples to compare the exponential decay factor in the loss, i.e. (image, text, length). By introducing the decay factor of the input granularity into the loss function, the inadaptability of the original base large model in processing fine-grained text is alleviated, thereby improving the stability and overall performance of the model training; S2: Add learnable prefixes to the large language model embedding layer and add a new template learning layer, such as Figure 8 As shown; S3: According to and The calculation formula modifies the original contrast loss function; S4: Contrastive loss function optimization, including: Freeze the large language model and visual encoder, set the hyperparameter T used to control the attenuation strength to t1, and randomly initialize and fine-tune the prefix template learning layer; Freeze the large language model, keep the hyperparameter T unchanged, and fine-tune the visual encoder and prefix template learning layer; Freeze the large language model and set the hyperparameter T to infinity. , fine-tuning the visual encoder and prefix template learning layer.

[0037] Through the above methods, the retrieval accuracy of the multimodal large model in cross-modal retrieval tasks for the human body is significantly improved, while the stability of the entire training process is enhanced.

[0038] On the other hand, this embodiment provides a multimodal human body model training system considering multi-granularity semantic alignment, including: A prefix construction module, used to construct prefixes for a large-scale training set by improving the network structure, and to add description-related prefixes to the data in the training set based on human features; The contrast loss improvement module is used to train the multimodal human body model in batches, pair each image with each text in each batch of training data, improve the contrast loss through similarity calculation and semantic relaxation, and optimize the visual and text modality encoder parameters.

[0039] Among them, the functional implementation of each module in the multimodal human body large model training system considering multi-granularity semantic alignment corresponds to the steps in the above-mentioned multimodal human body large model training method considering multi-granularity semantic alignment. Its functions and implementation process will not be repeated here one by one.

[0040] Finally, it should be noted that the above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

[0041] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims

1. A multimodal human body model training method considering multi-granularity semantic alignment, characterized in that: include: Prefix construction is performed on a large-scale training set by improving the network structure, and prefixes describing relevant data in the training set are added based on human body characteristics; The multimodal human body model is trained in batches, each image in each batch of training data is paired with each text, the contrast loss is improved by similarity calculation and semantic relaxation, and the parameters of the visual and text modality encoders are optimized.

2. The multimodal human body model training method considering multi-granularity semantic alignment according to claim 1, characterized in that: The prefix construction of the large-scale training set by improving the network structure includes: A prefix template learning layer is added between the original embedding layer and the large language model to construct the multimodal human body large model, learn prefix representations in an end-to-end manner, and automatically generate corresponding description-related prefixes for each data.

3. The multimodal human body model training method considering multi-granularity semantic alignment according to claim 2, characterized in that: The prefix construction of the large-scale training set further includes: Preprocessing the images and texts in the training set in the embedding layer, wherein the preprocessing of the images includes cleaning, labeling and normalization, and the preprocessing of the text includes pairing and format conversion, and at the same time, a certain position is reserved at the beginning of the text data for adding a prefix; The prefix template learning layer is based on a neural network structure, receives the text vector preprocessed by the embedding layer, generates a suitable prefix vector after neuron activation and weight adjustment, adds the prefix vector to the reserved prefix position, rewrites the input text, and generates a text vector with the prefix; After iterative training of the training set data, the prefix template learning layer realizes automatic generation of prefixes.

4. The multimodal human body model training method considering multi-granularity semantic alignment according to claim 3 is characterized in that: The adding of description-related prefixes to the data in the training set based on human body features further comprises: The text modality is prefix transformed, and the original text is rewritten according to the human body features through template combination. A prefix matching the description is added to the original text. The prefix matches the multi-granularity semantics of the image with the text data of each granularity, so that after multiple rounds of iterations, the descriptions of various granularities of the multimodal human body model collapse to the same embedding layer, avoiding the performance collapse of the multimodal human body model caused by the linear characteristics of multi-granularity semantics and contrast loss.

5. The multimodal human body model training method considering multi-granularity semantic alignment according to claim 1, characterized in that: The similarity calculation further includes: Pair each image in the batch with each text to generate an N×N similarity matrix, where N is the number of samples in the current batch, and each sample includes an image and its corresponding text; Calculate the similarity between each image and its corresponding text, and also calculate the similarity between the image and every other text in its batch; Maximize the similarity of correct pairs while minimizing the similarity of incorrect pairs.

6. The multimodal human body model training method considering multi-granularity semantic alignment according to claim 5, characterized in that: In the contrastive learning of the multimodal human body model, the loss includes image-to-text cross entropy loss and text-to-image cross entropy loss , expressed as: 。 7. The multimodal human body model training method considering multi-granularity semantic alignment according to claim 5, characterized in that: The improvement of the comparison loss by similarity calculation specifically includes: For a batch of training samples of a class, the image-to-text cross entropy loss The calculation formula is as follows: in, Indicates The predicted probability of image text matching.

8. The multimodal human body model training method considering multi-granularity semantic alignment according to claim 7, characterized in that: The said Image text matching prediction probability It is calculated by softmax, and the calculation formula is as follows: in, Represents the feature vector With the eigenvector The inner product of The calculation formula is as follows: in, Indicates image feature vector, Indicates A text feature vector.

9. The multimodal human body model training method considering multi-granularity semantic alignment according to claim 8, characterized in that: The improvement of contrast loss by semantic relaxation specifically includes: A new length field is added to the original training sample, which is used as the exponential decay factor in the contrast loss. In the early stage of model training, the granularity is approximately characterized by the text length, and different degrees of relaxation are performed for different granularity semantics in the original contrast loss function. Exponential decay is introduced into the calculation formula, and the calculation formula is expressed as follows: in, represents the actual input length of the current text. The hyperparameter T is used to control the attenuation strength. The prefix template learning layer is randomly initialized and fine-tuned. When T is set to infinity, .

10. A multimodal human body model training system considering multi-granularity semantic alignment, characterized in that: include: A prefix construction module is used to construct prefixes for a large-scale training set by improving the network structure, and to add description-related prefixes to the data in the training set based on human body features; The contrast loss improvement module is used to train the multimodal human body model in batches, pair each image with each text in each batch of training data, improve the contrast loss through similarity calculation and semantic relaxation, and optimize the visual and text modality encoder parameters.

Images