Image-text model training method and system based on image-text global information

Abstract

The invention provides an image-text model training method and system based on image-text global information. The method comprises the steps of obtaining original text information, a multi-image normal form and image information; inputting the image information into a multi-image normal form encoder in a preset image-text model to generate associated features of a plurality of images; inputting the multi-image normal form and the image information into an image-text semantic converter in a preset image-text model to obtain converted image features; and inputting the text information, the association features of the plurality of images and the converted image features into a large language model in a preset image-text model, training the large language model, completing association identification between the plurality of images and the video image-text, and obtaining a trained multi-modal image-text model. According to the method, the problems that data barriers exist between existing image-text modes, and deep collaborative understanding and processing are difficult are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal large models, and in particular to a method and system for training a graphic-text model based on global graphic-text information. Background Art

[0002] Image and sentence matching is one of the fundamental tasks in the field of vision and language. The goal of this cross-modal matching task is to accurately measure the visual semantic similarity between images and sentences. Despite significant progress in recent years, it remains a challenging problem because it requires understanding of language semantics, visual content, and cross-modal relations and alignments. Driven by the success of deep learning, the mainstream has shifted to modality-specific deep feature learning, for example, learning convolutional neural networks for images and recurrent neural networks for sentences. For visual text content understanding, many mature deep backbone models have been developed, such as VGG, ResNet, and Bert, which have improved the performance of various tasks to a significant level. Most of the existing successful image-text matching models can be divided into two categories according to the granularity of modality representation: global embedding methods and local reasoning methods. The former first embeds the entire image and sentence into a joint embedding space and then calculates the visual semantic similarity. However, methods that only use global alignment always show limited performance because text descriptions usually contain more fine-grained image details, which are easily smoothed by global alignment, resulting in insufficient modeling of fine-grained details. The latter approach first extracts a more fine-grained representation of local feature fragments (e.g., visual regions and words), and then aggregates local similarities to infer image-text alignment. Most existing methods achieve this goal by representing shared semantics as a weighted combination of all feature fragments (image regions or text words), where feature fragments related to the shared semantics receive more attention and those that are not receive less attention. However, while relevant factors contribute more to the shared semantics, irrelevant factors will more or less disturb the shared semantics, resulting in semantic misalignment at the relevant stage.

[0003] In the prior art, image information is encoded separately, and there is a lack of information interaction between images, resulting in the lack of key information in the encoding features, such as the contrast relationship between images, the development trend between images, and the connection between different frames in the video; it is impossible to accurately filter redundant information according to the input text instructions, resulting in the inability to correctly answer questions due to the focus shift, or the inability to achieve an accurate and reasonable reasoning path. Through the LoRA training technology, additional parameters for image modalities are specially trained, and there are still barriers between the image and text modalities, and in-depth collaborative understanding cannot be achieved; the training of the current model is highly dependent on image and text alignment data, and a lot of manpower and time are required on the data. The training of large image and text models cannot use massive existing data like language models, and model training can be performed without a lot of manual processing. For example, the language model training paradigm of "next word prediction" can be trained directly without manual annotation, while the image and text model needs to rely on manually annotated "image and text pair" training data. Summary of the invention

[0004] The present invention provides a method and system for training a graphic model based on global graphic information, so as to solve the problem that data barriers exist between existing graphic and text modalities and it is difficult to conduct in-depth collaborative understanding and processing.

[0005] The present invention provides a method for training a graphic model based on global graphic information, comprising: Obtain original text information, multi-image paradigm and image information; Inputting the image information into a multi-image paradigm encoder in a preset image-text model to generate correlation features of multiple images; Inputting the multi-image paradigm and image information into an image-text semantic converter in a preset image-text model to obtain converted image features; The text information, the associated features of the multiple images and the converted image features are input into a large language model in a preset image-text model, the large language model is trained, the association recognition between the multiple images and the video images and texts is completed, and a trained multimodal image-text model is obtained.

[0006] According to a method for training a graph-text model based on global graph-text information provided by the present invention, the image information is input into a multi-image paradigm encoder in a preset graph-text model to generate correlation features of multiple images, specifically comprising: The multi-image paradigm encoder captures the association, contrast and progression between images; Initializing the parameters of the multi-image paradigm encoder, and generating the associated features of multiple images through the multi-image paradigm encoder.

[0007] According to a method for training a graphic-text model based on global graphic-text information provided by the present invention, the preset graphic-text model obtains a global graphic-text semantic vector based on original text information and image information through a first forward propagation.

[0008] According to a method for training a picture-text model based on picture-text global information provided by the present invention, after obtaining the picture-text global semantic vector through the first forward propagation, the method further includes: Based on the image-text global semantic vector, the image-text model performs a second forward propagation to remove redundant information from the original input text information and image information.

[0009] According to a method for training a text-image model based on global text-image information provided by the present invention, the text information, the associated features of the multiple images and the converted image features are input into a large language model in a preset text-image model, the large language model is trained, the associated recognition between multiple images and video text-image is completed, and the trained multimodal text-image model is obtained, which specifically includes: Deleting input data of different modalities in a plurality of ways based on the input text information, the associated features of the plurality of images and the converted image features; By deleting input data of different modalities, the image-text model is forced to extract the key information required for the task from the global semantic vector of the image and text and the associated features of multiple images.

[0010] According to a method for training a graphic model based on global graphic information provided by the present invention, the input data of different modalities are deleted respectively in a plurality of ways based on the input text information, the associated features of the plurality of images and the converted image features, specifically including: Set the training goal of deleting text information to strengthen the graph-text model's understanding of text instructions, so that the graph-text model can mine text instruction information from the global semantic vector of the graph and text. This requires constructing training data without text instructions. Setting the training goal of deleting image information to strengthen the memory of image information in the graph-text model, so that the graph-text model can mine image instruction information from the global semantic vector of the graph and the associated features of multiple images, requires constructing training data without image input; Set the training goal of deleting multi-image association information to enhance the graph-text model's ability to extract multi-image association features, so that the graph-text model can mine multi-image association information from the global semantic vector of the graph and text without generating additional data.

[0011] The present invention also provides a graphic model training system based on graphic global information, the system comprising: Data acquisition module, used to obtain original text information, multi-image paradigm and image information; A correlation feature generation module, used for inputting the image information into a multi-image paradigm encoder in a preset image-text model to generate correlation features of multiple images; A feature conversion module, used for inputting the multi-image paradigm and image information into a picture-text semantic converter in a preset picture-text model to obtain converted image features; The model training module is used to input the text information, the associated features of the multiple images and the converted image features into a large language model in a preset image-text model, train the large language model, complete the association recognition between multiple images and video images and texts, and obtain a trained multimodal image-text model.

[0012] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for training a graphic model based on global graphic information as described above is implemented.

[0013] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described methods for training a graphic model based on global graphic information.

[0014] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned graphic-text model training methods based on graphic-text global information.

[0015] The present invention provides a method and system for training a graph-text model based on global information of graphs and texts. Through a graph-text model structure based on learning of global information of graphs and texts and feature paradigms between images, a training strategy for learning global information of graphs and texts and feature paradigms between images is designed. The strategy includes a new pre-training target that is different from the target of image description generation, as well as a task fine-tuning training target; through an automated data batch generation method, existing "graph-text pair" data is collected, including two forms of "text-image" data pairs and "video-image" data pairs. Based on the existing "graph-text pair" data, a large amount of data in the form of interleaved graphs and texts is automatically generated without manual annotation, and finally model pre-training and task fine-tuning are performed based on these data to break the data barriers between graph-text modalities and achieve in-depth collaborative understanding and processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 It is a flow chart of a method for training a graphic model based on global graphic information provided by the present invention.

[0018] Figure 2 It is the information graphic model structure diagram provided by the present invention.

[0019] Figure 3 It is a schematic diagram of module connection of a graphic-text model training system based on graphic-text global information provided by the present invention.

[0020] Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention.

[0021] Figure numerals: 110: data acquisition module; 120: associated feature generation module; 130: feature conversion module; 140: model training module; 410: processor; 420: communication interface; 430: memory; 440: communication bus. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] Combine the following Figure 1 The invention describes a method for training a graphic-text model based on global graphic-text information, including: step 100, obtaining original text information, multi-graphic paradigm and image information.

[0024] In the present invention, the association between video and image data is identified based on the acquired text information, multi-image paradigm and image information through a preset multi-image paradigm encoder, image-text semantic converter and large language model, which can be used for reasoning tasks in multi-image and video scenes.

[0025] Step 200: Input the image information into a multi-image paradigm encoder in a preset image-text model to generate correlation features of multiple images.

[0026] Specifically, the multi-image paradigm encoder captures the association, contrast and progressive relationship between images; Initializing the parameters of the multi-image paradigm encoder, and generating the associated features of multiple images through the multi-image paradigm encoder.

[0027] In the present invention, the multi-image paradigm encoder captures the association, contrast relationship, and progressive relationship between images. For multi-image scenes and video scenes (involving multiple image frames), the encoding of different images is independent of each other, and their feature vectors do not contain other image information. Therefore, there is a problem of missing and inaccurate association information between images, so it is necessary to learn the multi-image paradigm additionally.

[0028] Step 300: Input the multi-image paradigm and image information into a picture-text semantic converter in a preset picture-text model to obtain converted image features.

[0029] In the present invention, a preset image-text model obtains an image-text global semantic vector based on the original text information and image information through a first forward propagation; based on the image-text global semantic vector, the image-text model performs a second forward propagation to remove redundant information from the original input text information and image information.

[0030] Specifically, the global semantic vector of the image and text is the image and text model based on Figure 2 The global instructions learned from the three types of input (text information, multi-image paradigm, image information) shown in the figure contain the semantic information of the image and text modality and the current task goal information, guiding the model to make the correct response.

[0031] The overall graph-text model is denoted as M, and the large language model is denoted as M L , the multi-image paradigm encoder is denoted as M p , the image-text semantic converter is denoted as Q. The image feature is represented as I n (involving N images, n=1,…,N, for video data, N represents the N frames required for encoding), I p is the correlation feature of multiple images / frames. Multi-image paradigm encoder M p Transformer, CNN or hybrid structure can be used. p The calculation does not involve C, and the specific calculation method is as follows.

[0032] .

[0033] The global semantic vector of the image and text is obtained from the image and text model M, and this vector is recorded as C. This case needs to go through two forward propagations of the image and text model. The first one is to obtain the global semantic vector of the image and text, and the second one is to use the global semantic vector of the image and text to remove redundant information from the original image and text input, so that the model can accurately focus on the information that is effective for answering. In the first forward propagation, the global semantic vector C of the image and text does not need to be considered. Only C and the corresponding feature representation are added in the second forward propagation. Figure 2 In the figure, the global semantic vector of the image and text is marked with a dotted line. In the present invention, the word embedding vector of the text is represented by T, I' nis the image feature after the image-text semantic converter. In the first forward propagation, the image-text global semantic vector C is calculated as follows.

[0034] .

[0035] In the second forward propagation, the calculation of image transformation features needs to take C into account.

[0036] .

[0037] Finally, the output y of the graph-text model is calculated as follows.

[0038] .

[0039] Step 400: input the text information, the associated features of the multiple images and the converted image features into a large language model in a preset image-text model, train the large language model, complete the association recognition between multiple images and video images and text, and obtain a trained multimodal image-text model.

[0040] Specifically, the method comprises: deleting input data of different modes in a plurality of ways based on the input text information, the associated features of the plurality of images and the converted image features; By deleting input data of different modalities, the image-text model is forced to extract the key information required for the task from the global semantic vector of the image and text and the associated features of multiple images.

[0041] Wherein, based on the input text information, the associated features of the multiple images and the converted image features, the input data of different modalities are deleted respectively in a variety of ways, specifically including: Set the training goal of deleting text information to strengthen the graph-text model's understanding of text instructions, so that the graph-text model can mine text instruction information from the global semantic vector of the graph and text. This requires constructing training data without text instructions. Setting the training goal of deleting image information to strengthen the memory of image information in the graph-text model, so that the graph-text model can mine image instruction information from the global semantic vector of the graph and the associated features of multiple images, requires constructing training data without image input; Set the training goal of deleting multi-image association information to enhance the graph-text model's ability to extract multi-image association features, so that the graph-text model can mine multi-image association information from the global semantic vector of the graph and text without generating additional data.

[0042] In the present invention, the multi-image paradigm encoder M p Initialize the parameters of M L Able to initially learn to use the global semantic vector C of images and text and the associated features I of multiple images / frames p, perform semantic alignment training on this information. The loss function in the pre-training stage is Consistent with the loss function of the image description generation task, cross entropy loss is used.

[0043] Task fine-tuning can optimize M according to specific tasks. p The learning process of the association of multiple images / frames; and deepening M L The global semantic vector C of the image and text and the associated features I of multiple images / frames p At the same time, with the help of this information, the deep understanding of multi-image and video information can be completed, and high-precision reasoning can be achieved. At this time, the present invention performs special processing on the training data, and deletes the input data of different modes in three ways, forcing the model to learn from the global semantic vector C of the image and text and the associated features I of multiple images / frames. p Extract the key information required for the task. For the same original input data, there are 4 different training objectives: Training objectives for original tasks, such as image description generation, image question answering, image classification, etc. In order to remove the training goal of text information, strengthen the model's understanding of text instructions, and let the model learn to mine text instruction information from C, it is necessary to construct training data without text instructions; To delete the training goal of image information, strengthen the model's memory of image information, and let the model learn from C and I p To mine image instruction information, it is necessary to construct training data without image input; In order to delete the training target of multi-graph correlation information and strengthen the model's ability to extract multi-graph correlation features, the model can learn to mine multi-graph correlation information from C without generating additional data.

[0044] During batch training, the same batch of data completes the calculation of the above four loss functions and then updates the parameters uniformly. Four types of losses need to be considered at the same time. Among them, the loss of deleting different modal information needs to calculate the absolute value of the difference between it and the original data loss. The final calculation method is as follows.

[0045] .

[0046] During the pre-training and fine-tuning stages, the loss and gradient are not calculated in the first forward propagation, and the loss and gradient are only calculated after the second forward propagation (using C).

[0047] In addition, in the present invention, since the image and text content are deleted during training, the following data generation mechanism is designed. , this case is based on the proportion Delete the input text % of text, where Probability for.

[0048] .

[0049] For the loss function , this case is based on the proportion Delete the input image %,in Probability for.

[0050] .

[0051] Among them, the number of deleted images is rounded up, that is, the number of deleted images is For example, an input contains 3 pictures. When (i.e. 50% of the images are deleted), you need to delete pictures.

[0052] In the training process of the graph model, four times or more of the original training data can be used, including four different information deletion methods, and a method of proportional deletion of graph data based on different probabilities. The number of training steps of the model in this case is at least four times the number of training steps of the usual model, which improves the data scale and data utilization without manual labeling.

[0053] The training of the image-text model in the present invention is divided into pre-training for information alignment and task fine-tuning for actual task optimization. In the pre-training stage, only the multi-image paradigm encoder M is updated. p Parameters, M p The parameters are initialized randomly or to zero, and a larger learning rate is required for updating. The recommended learning rate range is [0.001, 0.0001]. p , Q and M L Optimize the parameters. p All parameters of M and Q need to be optimized. L The parameters of M only need to optimize the parameters of the self-attention layer, L The other parameters of M are fixed. p After training, a certain ability of multi-image paradigm understanding, extraction and encoding has been obtained. In this stage, all parameters that need to be optimized use the same small learning rate to prevent overfitting. A learning rate less than or slightly greater than 0.00001 can be used.

[0054] Through the graph-text model training method based on the global information of graphs and texts provided by the present invention, a training strategy for learning the global information of graphs and texts and the feature paradigm between images is designed through a graph-text model structure based on the learning of the global information of graphs and texts and the feature paradigm between images. The strategy includes a new pre-training target that is different from the target of image description generation, as well as a task fine-tuning training target; through an automated data batch generation method, existing "graph-text pair" data is collected, including two forms of "text-image" data pairs and "video-image" data pairs. Based on the existing "graph-text pair" data, a large amount of data in the form of interleaved graphs and texts is automatically generated without manual annotation, and finally model pre-training and task fine-tuning are performed based on these data to break the data barriers between graph-text modalities and achieve in-depth collaborative understanding and processing.

[0055] refer to Figure 3 The present invention also discloses a graphic model training system based on graphic global information, the system comprising: The data acquisition module 110 is used to acquire original text information, multi-image paradigm and image information; The associated feature generation module 120 is used to input the image information into a multi-image paradigm encoder in a preset image-text model to generate associated features of multiple images; A feature conversion module 130, configured to input the multi-image paradigm and image information into an image-text semantic converter in a preset image-text model to obtain converted image features; The model training module 140 is used to input the text information, the associated features of the multiple images and the converted image features into a large language model in a preset image-text model, train the large language model, complete the association recognition between multiple images and video images and texts, and obtain a trained multimodal image-text model.

[0056] The image information is input into a multi-image paradigm encoder in a preset image-text model to generate correlation features of multiple images, specifically including: The multi-image paradigm encoder captures the association, contrast and progression between images; Initializing the parameters of the multi-image paradigm encoder, and generating the associated features of multiple images through the multi-image paradigm encoder.

[0057] The preset image-text model obtains the global semantic vector of the image-text based on the original text information and image information through the first forward propagation.

[0058] After the first forward propagation to obtain the global semantic vector of the image and text, it also includes: Based on the image-text global semantic vector, the image-text model performs a second forward propagation to remove redundant information from the original input text information and image information.

[0059] Inputting the text information, the associated features of the multiple images and the converted image features into a large language model in a preset image-text model, training the large language model, completing the association recognition between the multiple images and the video image-text, and obtaining a trained multimodal image-text model, specifically includes: Deleting input data of different modalities in a plurality of ways based on the input text information, the associated features of the plurality of images and the converted image features; By deleting input data of different modalities, the image-text model is forced to extract the key information required for the task from the global semantic vector of the image and text and the associated features of multiple images.

[0060] Based on the input text information, the associated features of the multiple images and the converted image features, the input data of different modalities are deleted in a variety of ways, including: Set the training goal of deleting text information to strengthen the graph-text model's understanding of text instructions, so that the graph-text model can mine text instruction information from the global semantic vector of the graph and text. This requires constructing training data without text instructions. Setting the training goal of deleting image information to strengthen the memory of image information in the graph-text model, so that the graph-text model can mine image instruction information from the global semantic vector of the graph and the associated features of multiple images, requires constructing training data without image input; Set the training goal of deleting multi-image association information to enhance the graph-text model's ability to extract multi-image association features, so that the graph-text model can mine multi-image association information from the global semantic vector of the graph and text without generating additional data.

[0061] Through the graphic model training system based on global information of graphics and text provided by the present invention, a training strategy for learning global information of graphics and text and feature paradigm between images is designed through a graphic model structure based on learning global information of graphics and text and feature paradigm between images. The strategy includes a new pre-training target that is different from the target of image description generation, as well as a task fine-tuning training target; through an automated data batch generation method, existing "graphic-text pair" data is collected, including two forms of "text-image" data pairs and "video-image" data pairs. Based on the existing "graphic-text pair" data, a large amount of data in the form of graphics and text interleaving is automatically generated without manual annotation, and finally model pre-training and task fine-tuning are performed based on these data to break the data barriers between graphic and text modalities and achieve in-depth collaborative understanding and processing.

[0062] Figure 4 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 4As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430 and a communication bus 440, wherein the processor 410, the communication interface 420 and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute a method for training a graph-text model based on global graph-text information, the method comprising: obtaining original text information, multi-graph paradigm and image information; inputting the image information into a multi-graph paradigm encoder in a preset graph-text model to generate associated features of multiple images; inputting the multi-graph paradigm and image information into a graph-text semantic converter in a preset graph-text model to obtain converted image features; inputting the text information, the associated features of the multiple images and the converted image features into a large language model in a preset graph-text model, training the large language model, completing the association recognition between the multi-graph and the video graph-text, and obtaining a trained multimodal graph-text model.

[0063] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0064] On the other hand, the present invention also provides a computer program product, which includes a computer program, and the computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the image-text model training method based on the global information of the image and text provided by the above methods, and the method includes: obtaining original text information, multi-image paradigm and image information; inputting the image information into the multi-image paradigm encoder in the preset image-text model to generate the associated features of multiple images; inputting the multi-image paradigm and image information into the image-text semantic converter in the preset image-text model to obtain the converted image features; inputting the text information, the associated features of the multiple images and the converted image features into the large language model in the preset image-text model, training the large language model, completing the association recognition between the multiple images and the video images and texts, and obtaining the trained multimodal image-text model.

[0065] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the graph-text model training method based on global graph-text information provided by the above-mentioned methods, the method comprising: obtaining original text information, multi-image paradigm and image information; inputting the image information into a multi-image paradigm encoder in a preset graph-text model to generate associated features of multiple images; inputting the multi-image paradigm and image information into a graph-text semantic converter in a preset graph-text model to obtain converted image features; inputting the text information, the associated features of the multiple images and the converted image features into a large language model in a preset graph-text model, training the large language model, completing the association recognition between multiple images and video graphs and texts, and obtaining a trained multimodal graph-text model.

[0066] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0067] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0068] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for training a graph-text model based on global graph-text information, characterized in that: include: Obtain original text information, multi-image paradigm and image information; Inputting the image information into a multi-image paradigm encoder in a preset image-text model to generate correlation features of multiple images; Inputting the multi-image paradigm and image information into an image-text semantic converter in a preset image-text model to obtain converted image features; The text information, the associated features of the multiple images and the converted image features are input into a large language model in a preset image-text model, the large language model is trained, the association recognition between the multiple images and the video images and texts is completed, and a trained multimodal image-text model is obtained.

2. The method for training a graphic model based on global graphic information according to claim 1, characterized in that: The step of inputting the image information into a multi-image paradigm encoder in a preset image-text model to generate correlation features of multiple images specifically includes: The multi-image paradigm encoder captures the association, contrast and progression between images; Initializing the parameters of the multi-image paradigm encoder, and generating the associated features of multiple images through the multi-image paradigm encoder.

3. The method for training a graphic-text model based on global graphic-text information according to claim 1, characterized in that: The preset image-text model obtains the image-text global semantic vector based on the original text information and image information through the first forward propagation.

4. The method for training a graphic model based on global graphic information according to claim 3, characterized in that: After the first forward propagation to obtain the global semantic vector of the image and text, the method further includes: Based on the image-text global semantic vector, the image-text model performs a second forward propagation to remove redundant information from the original input text information and image information.

5. The method for training a graphic model based on global graphic information according to claim 1, characterized in that: The step of inputting the text information, the associated features of the multiple images, and the converted image features into a large language model in a preset image-text model, training the large language model, completing the association recognition between the multiple images and the video image-text, and obtaining a trained multimodal image-text model specifically includes: Deleting input data of different modalities in a plurality of ways based on the input text information, the associated features of the plurality of images and the converted image features; By deleting input data of different modalities, the image-text model is forced to extract the key information required for the task from the global semantic vector of the image and text and the associated features of multiple images.

6. The method for training a graphic-text model based on global graphic-text information according to claim 5, characterized in that: The method of deleting input data of different modes based on the input text information, the associated features of the multiple images and the converted image features adopts multiple methods, specifically including: Set the training goal of deleting text information to strengthen the graph-text model's understanding of text instructions, so that the graph-text model can mine text instruction information from the global semantic vector of the graph and text. This requires constructing training data without text instructions. Setting the training goal of deleting image information to strengthen the memory of image information in the graph-text model, so that the graph-text model can mine image instruction information from the global semantic vector of the graph and the associated features of multiple images, requires constructing training data without image input; Set the training goal of deleting multi-image association information to enhance the graph-text model's ability to extract multi-image association features, so that the graph-text model can mine multi-image association information from the global semantic vector of the graph and text without generating additional data.

7. A graphic model training system based on global graphic information, characterized in that: The system comprises: Data acquisition module, used to obtain original text information, multi-image paradigm and image information; A correlation feature generation module, used for inputting the image information into a multi-image paradigm encoder in a preset image-text model to generate correlation features of multiple images; A feature conversion module, used for inputting the multi-image paradigm and image information into a picture-text semantic converter in a preset picture-text model to obtain converted image features; The model training module is used to input the text information, the associated features of the multiple images and the converted image features into a large language model in a preset image-text model, train the large language model, complete the association recognition between multiple images and video images and texts, and obtain a trained multimodal image-text model.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the image-text model training method based on image-text global information as described in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for training a graph-text model based on global graph-text information as described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for training a graph-text model based on global graph-text information as described in any one of claims 1 to 6 is implemented.

Images