Joint generation type pre-training-based large language model training method and device

Abstract

The invention provides a large language model training method and device based on joint generation type pre-training, and the method comprises the steps: obtaining a medical image and a description text, and carrying out the coding through a visual encoder, and obtaining a feature map; for each pixel code in the feature map, querying similar codes of the pixel codes from an original unified codebook of the multi-modal large language model to obtain a codebook map and a quantitative feature map; obtaining a query indication diagram, and performing similar code screening on the original unified codebook to obtain a screened updated codebook; and finally, performing attenuation updating through the model codebook of the multi-modal large language model to obtain a training updating codebook, and training the multi-modal large language model based on the training updating codebook. By means of the method and device, the defects that in the prior art, mapping learning from visual information to text information in the medical field becomes complex and difficult, information integration is not accurate enough, and consequently the applicability and the application effect of a multi-modal large language model in the medical field are limited are overcome.

Description

Technical Field

[0001] The present invention relates to the field of medical technology, and in particular to a training method and device for a large language model based on joint generative pre-training. Background Art

[0002] In the medical field, especially in surgery and diagnosis, automatically parsing medical images and generating relevant text descriptions has important application value. This automation technology can not only significantly reduce the workload of doctors, but also improve the accuracy and efficiency of surgery and diagnosis. Traditional methods usually rely on data processing of a single modality (such as pure vision or pure text), which often shows its limitations when processing complex medical data.

[0003] At present, multimodal learning, that is, the simultaneous use of visual and textual data, has become a trend. However, the multimodal methods in the existing technology still face some challenges in medical applications. First, there are significant differences in the characteristics of medical images and related textual data, which makes it complicated and difficult to learn the mapping directly from vision to text. Secondly, although multimodal methods improve the knowledge fusion between modalities by jointly optimizing the visual encoder and text decoder, they usually fail to fully utilize the intrinsic connection between vision and text, resulting in insufficient in-depth and precise integration of information. In addition, multimodal learning methods often rely on pre-trained visual encoders and text decoders. These pre-trained models are usually trained on non-medical data, so their applicability and effectiveness in medical scenarios are limited. Summary of the invention

[0004] The present invention provides a training method and device for a large language model based on joint generative pre-training, which is used to solve the defects in the prior art that the mapping learning from visual information to text information of the multimodal large language model in the medical field becomes complicated and difficult, and the information integration is not accurate enough, resulting in the limitation of the applicability and application effect of the multimodal large language model in the medical field.

[0005] The present invention provides a training method for a large language model based on joint generative pre-training, the method comprising the following steps: Acquire a medical image and a corresponding description text, encode the medical image through a visual encoder, and obtain a feature map of the medical image; For each pixel code in the feature map, query similar codes of the pixel code from an original unified codebook of the multimodal large language model to obtain a codebook map and a quantized feature map corresponding to the codebook map; Acquire a query indication graph, and screen the original unified codebook for similar coding based on the query indication graph to obtain a screened updated codebook, wherein the query indication graph is determined based on the flattened quantized feature graph; The screening update codebook is attenuated and updated through the model codebook of the multimodal large language model to obtain a training update codebook, and the multimodal large language model is trained based on the training update codebook, wherein the model codebook is obtained by fusing the feature map and the text features of the description text.

[0006] In some embodiments, the querying of similar codes of the pixel codes from the original unified codebook of the multimodal large language model to obtain a codebook graph and a quantized feature graph corresponding to the codebook graph includes: Determining the Euclidean distance between the pixel code and each target code of an original unified codebook of a multimodal large language model; The target code with the smallest Euclidean distance is used as the similarity code corresponding to the pixel code; Combining similar codes corresponding to each pixel code to obtain a codebook graph; The expected value of the codebook image is used as the quantization feature image corresponding to the codebook image.

[0007] In some embodiments, obtaining the query indication graph includes: Flattening the quantized feature map to obtain a flattened feature map; For each subcode in the original unified codebook, when the subcode exists in the flattened feature map, marking the subcode with a used label; When the sub-code does not exist in the flattened feature map, marking the sub-code with an unused label; A query indication graph is constructed according to the used labels and the unused labels.

[0008] In some embodiments, the screening of similar codes in the original unified codebook based on the query indication graph to obtain a screened updated codebook includes: Obtaining a first attenuation ratio, and determining a difference between 1 and the first attenuation ratio as a second attenuation ratio; Weighting the original unified codebook by the first attenuation ratio to obtain a first weighted result; The flattened feature graph is screened by the query indication graph to obtain a screened feature graph, and the screened feature graph is weighted by the second attenuation ratio to obtain a second weighted result; The sum of the first weighted result and the second weighted result is used as a filter to update the codebook.

[0009] In some embodiments, the attenuation updating of the screening update codebook by using the model codebook of the multimodal large language model to obtain the training update codebook includes: weighting the screened updated codebook according to the first attenuation ratio to obtain a third weighted result; weighting the model codebook by a second attenuation ratio to obtain a fourth weighted result; The sum of the third weighted result and the fourth weighted result is used as a training updated codebook.

[0010] In some embodiments, the method further comprises: Calling a decoder to perform image decoding processing on the codebook image to obtain a reconstructed medical image of the medical image; The reconstructed medical image is encoded by the visual encoder to obtain a reconstructed feature map of the reconstructed medical image, wherein the reconstructed feature map is used to determine a reconstructed codebook map to update the original unified codebook.

[0011] The present invention also provides a training device for a large language model based on joint generative pre-training, the device comprising the following modules: An encoding module, used for acquiring a medical image and a corresponding description text, encoding the medical image through a visual encoder, and obtaining a feature map of the medical image; A query module is used to query similar codes of each pixel code in the feature map from an original unified codebook of the multimodal large language model to obtain a codebook map and a quantized feature map corresponding to the codebook map; A screening module, used to obtain a query indication graph, and screen the original unified codebook for similar coding based on the query indication graph to obtain a screened updated codebook, wherein the query indication graph is determined based on the flattened quantized feature graph; A training module is used to attenuate and update the screening update codebook through the model codebook of the multimodal large language model to obtain a training update codebook, and train the multimodal large language model based on the training update codebook, wherein the model codebook is obtained by fusing the feature map and the text features of the description text.

[0012] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the training method of a large language model based on joint generative pre-training as described in any one of the above is implemented.

[0013] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a training method for a large language model based on joint generative pre-training as described in any one of the above.

[0014] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned methods for training a large language model based on joint generative pre-training.

[0015] The present invention provides a method and device for training a large language model based on joint generative pre-training.

[0016] By designing an efficient visual encoder and a unified codebook strategy, we can optimize the feature representation of visual images and text data and achieve in-depth integration of cross-modal knowledge. This solves the problem of coordination of visual and text modal knowledge in the medical field and precise integration of multimodal knowledge. This enables the large language model to naturally and comprehensively understand and integrate medical images and text data, thereby improving the applicability and application effect of the large language model in medical application fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced one by one below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 It is a flowchart of the training method of a large language model based on joint generative pre-training provided by the present invention.

[0019] Figure 2 It is a schematic diagram of the principle of the training method of the large language model based on joint generative pre-training provided by the present invention.

[0020] Figure 3 It is a structural schematic diagram of a training device for a large language model based on joint generative pre-training provided by the present invention.

[0021] Figure 4 It is a schematic diagram of the physical structure of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] Combine the following Figure 1-Figure 2The present invention describes a training method for a large language model based on joint generative pre-training. Figure 1 is a flow chart of the training method of a large language model based on joint generative pre-training provided by the present invention, such as Figure 1 As shown, the method includes the following steps 101 to 104.

[0024] Step 101: obtain a medical image and a corresponding description text, encode the medical image through a visual encoder, and obtain a feature map of the medical image.

[0025] First, a large number of medical images I can be collected during pathological analysis and clinical practice. These medical images are generally used to display the patient's pathological condition or clinical characteristics, and each medical image needs to collect a corresponding description text T. The description text T is generally used to describe the medical image, that is, to describe the patient's pathological condition or clinical characteristics.

[0026] The next step is to extract features from medical images and description text. Figure 2 , Figure 2 Schematic diagram of the principle of the training method of the large language model based on joint generative pre-training provided by the present invention, such as Figure 2 As shown, for medical images, the medical image I is encoded by the visual encoder E to obtain the feature map of the medical image , and for the description text T, the text encoder can be called to encode and obtain the text features of the description text. The visual encoder and text encoder here can use the visual text comparison pre-training model (CLIP), or other image encoders or text encoders, such as the Transformer model, which is not limited here.

[0027] Since the feature dimensions of the feature maps of medical images and text features are different, they cannot be directly sent to the multimodal large language model for prediction. In order to enable the multimodal large language model to learn the fusion features of vision and text, the embodiment of the present invention designs a unified codebook C, which can be regarded as a pre-built temporary feature database. This feature database is continuously updated by extracting the feature maps and text features of medical images each time and storing them therein. This original unified codebook is pre-built based on the fusion of the feature maps of historical medical images and the text features of the corresponding descriptive text. The fusion process can be to sequentially scan the feature map to form a one-dimensional image sequence feature, and then merge it with the text feature.

[0028] Step 102: for each pixel code in the feature map, query similar codes of the pixel code from the original unified codebook of the multimodal large language model to obtain a codebook map and a quantized feature map corresponding to the codebook map.

[0029] Here, for the feature map For each pixel code z in , query the similar code Q(z; C) of the pixel code z from the original unified codebook C of the multimodal large language model to obtain the codebook graph And the quantized feature map corresponding to the codebook map The specific process is: determine the Euclidean distance e between the pixel code z and each target code of the original unified codebook C of the multimodal large language model, and then use the target code with the smallest Euclidean distance as the similarity code e(k) corresponding to the pixel code z. The calculation formula is expressed as the following formula (1): (1)

[0030] In the above formula (1), z represents the feature map Each pixel in is encoded, K represents the total number of target codes in the original unified codebook C, represents the kth target code, Represents the Euclidean distance calculation.

[0031] After determining the similar code corresponding to each pixel code z, the similar code corresponding to each pixel code z is Combine to get the codebook graph ,Right now , where i and j represent pixel code z corresponding to the feature map The coordinates in the codebook map The expected value E( ) as the quantized feature map corresponding to the codebook map .

[0032] Step 103: Obtain a query indication graph, and screen the original unified codebook for similar codes based on the query indication graph to obtain a screened updated codebook.

[0033] First, the query indication graph MI is obtained. The process of constructing the query indication graph is as follows: first, the quantized feature graph Perform flattening to obtain a flattened feature map Z. Quantized feature map Since it is two-dimensional, the quantized feature map is processed by flattening , mapping it into one dimension, so as to facilitate alignment with the original unified codebook. The query indicator map MI is used to record whether each code in the original unified codebook C exists in the flattened feature map Z. Therefore, after obtaining the flattened feature map, for each subcode in the original unified codebook C, when the subcode exists in the flattened feature map Z, it means that the subcode has been used, and the subcode is marked with a used label, for example, marked as 1. When the subcode does not exist in the flattened feature map, it means that the subcode has been used, and the subcode is marked with an unused label, for example, marked as 0, so that the used codes can be filtered out from the original unified codebook according to the query indicator map MI.

[0034] Next, based on the query indication graph, the original unified codebook is screened for similar codes to obtain a screened updated codebook. In the embodiment of the present invention, the screening and updating process can be performed by combining an exponential sliding average method with a preset first attenuation ratio Specifically, the first attenuation ratio is obtained , and determine the first decay ratio of 1 The difference is the second attenuation ratio ( ); through the first attenuation ratio The original unified codebook C is weighted to obtain the first weighted result; the flattened feature map Z is screened by querying the indicator map MI to obtain the screened feature map (MI*Z). The screening process is the process of multiplying the feature map with the codebook by the feature vector. And through the second attenuation ratio Weight the screening feature map MI*Z to obtain a second weighted result; finally, the sum of the first weighted result and the second weighted result is used as the screening update codebook , expressed as formula (2): (2)

[0035] In the above formula (2), MI represents the query indicator map, Z represents the flattened feature map, represents the original unified codebook, Indicates the filtering and updating of the codebook, * indicates the multiplication of feature vectors, Indicates the first attenuation ratio.

[0036] Step 104: perform attenuation update on the screening update codebook through the model codebook of the multimodal large language model to obtain a training update codebook, and train the multimodal large language model based on the training update codebook.

[0037] In order to ensure that the feature map of the medical image is close to the text feature, it is convenient to integrate the visual feature and the text feature. Therefore, the embodiment of the present application regularly updates the original unified codebook through the feature map and the text feature. The original unified codebook here is the screened updated codebook obtained by screening the query indication map, so it is also recorded as C. The updating method is to use the model codebook of the multimodal large language model Perform attenuation update on the screening update codebook C to obtain the updated training update codebook , where the model codebook of the multimodal large language model is It is obtained by fusing the feature map Z and the text features of the description text T, and the attenuation update method still uses the exponential sliding average method.

[0038] Specifically, according to the first attenuation ratio The screened updated codebook C is weighted to obtain a third weighted result; the second attenuation ratio Model Codebook weighted to obtain a fourth weighted result; the sum of the third weighted result and the fourth weighted result is used as the training update codebook , expressed as the following formula (3):

[0039] In the above formula (3), represents the filtering and updating codebook, represents the updated training codebook, Represents the model codebook of the multimodal large language model, Indicates the first attenuation ratio.

[0040] Finally, update the codebook based on training Train a multimodal large language model. In the actual training process, the process of updating the training codebook is also the training process of the multimodal large language model.

[0041] In the subsequent training process, the training updated codebook is used as the original unified codebook for the next training round, and the above steps 101 to 104 are continued to be performed using the medical images and corresponding description texts of the next round to update the original unified codebook, thereby continuously enabling the large language model to effectively learn the visual feature map and text features. Through the updated original unified codebook, the visual features and text features can be aligned and can be perceived by the multimodal large language model, so that the multimodal large language model can more effectively integrate and understand the visual features and text features, thereby automatically parsing and generating accurate text descriptions or making text answers for medical decision-making.

[0042] In some embodiments, Figure 2As shown, in the process of updating the original unified encoding and training the multimodal large language model, the decoder (such as Figure 2 VQ-GAN decoder in the codebook Image decoding is performed to obtain a reconstructed medical image of the medical image I. Feature decoding can be used as a feedback process of feature extraction or feature fusion. According to the codebook composed of similar codes similar to the medical image, , it can be feature decoded to reshape it into an image. The reshaped medical image can be compared with the original medical image I. For example, some loss functions can be constructed and fed back to the visual encoder and the multimodal large language model to verify the effectiveness of the visual encoder and the original unified codebook update.

[0043] In some embodiments, the reshaped medical image can also add corresponding text descriptions as training data for the next round. Specifically, the reshaped medical image is encoded by a visual encoder to obtain a reshaped feature map of the reshaped medical image, and the reshaped feature map can be used to determine the reshaped codebook map, that is, from the original unified codebook C of the multimodal large language model, query the similar encoding corresponding to the reshaped feature map to obtain the codebook map and the quantized reshaped feature map corresponding to the reshaped codebook map, and then screen the similar encodings, and finally update the original unified codebook C to help the training of the multimodal large language model. The update process can refer to steps 103 to 104, which will not be repeated here.

[0044] The multimodal large language model obtained after training can be applied to computer-assisted surgery systems or medical systems, and can be applied to any type of lesions and surgical operations, including but not limited to highly variable surgical types, to meet diverse clinical requirements. By acquiring medical images related to surgical pathology and the corresponding image description texts in real time and making text responses in a timely manner, it can help output medical decisions and provide doctors with more accurate diagnosis and treatment support.

[0045] The training method of the large language model based on joint generative pre-training provided by the present invention can optimize the feature representation of visual images and text data, realize the in-depth integration of cross-modal knowledge, and solve the problem of the coordination of visual and text modal knowledge in the medical field and the precise integration of multimodal knowledge by designing an efficient visual encoder and a unified codebook strategy. In this way, the multimodal large language model can naturally and comprehensively understand and integrate medical images and text data. Furthermore, the present invention trains the multimodal large language model by using medical images and corresponding texts as training data, thereby improving the applicability and application effect of the model in the field of medical applications.

[0046] The following is a description of the training device for the large language model based on joint generative pre-training provided by the present invention. The training device for the large language model based on joint generative pre-training described below and the training method for the large language model based on joint generative pre-training described above can refer to each other.

[0047] See also Figure 3 , Figure 3 is a schematic diagram of the structure of a training device for a large language model based on joint generative pre-training provided by the present invention, such as Figure 3 As shown, the training device of the large language model based on joint generative pre-training includes an encoding module 301, a query module 302, a screening module 303, and a training module 304. Among them, the encoding module 301 is used to obtain a medical image and a corresponding description text, encode the medical image through a visual encoder, and obtain a feature map of the medical image; the query module 302 is used to query the similar code of the pixel code from the original unified codebook of the multimodal large language model for each pixel code in the feature map, and obtain a codebook map and a quantized feature map corresponding to the codebook map; the screening module 303 is used to obtain a query indication map, and screen the original unified codebook for similar codes based on the query indication map to obtain a screened update codebook, wherein the query indication map is determined based on the flattened quantized feature map; the training module 304 is used to attenuate and update the screened update codebook through the model codebook of the multimodal large language model to obtain a training update codebook, and train the multimodal large language model based on the training update codebook, wherein the model codebook is obtained by fusing the feature map and the text features of the description text.

[0048] Figure 4 is a schematic diagram of the physical structure of an electronic device provided by the present invention, such as Figure 4As shown, the electronic device may include: a processor (processor) 410 , a communication interface (Communications Interface) 420 , a memory (memory) 430 and a communication bus 440 , wherein the processor 410 , the communication interface 420 , and the memory 430 communicate with each other through the communication bus 440 . The processor 410 can call the logic instructions in the memory 430 to execute a training method for a large language model based on joint generative pre-training, the method comprising: obtaining a medical image and a corresponding description text, encoding the medical image through a visual encoder to obtain a feature map of the medical image; for each pixel code in the feature map, querying the similar code of the pixel code from the original unified codebook of the multimodal large language model to obtain a codebook map and a quantized feature map corresponding to the codebook map; obtaining a query indication map, and filtering the original unified codebook for similar codes based on the query indication map to obtain a filtered update codebook, wherein the query indication map is determined based on the flattened quantized feature map; attenuating and updating the filtered update codebook through the model codebook of the multimodal large language model to obtain a training update codebook, and training the multimodal large language model based on the training update codebook, wherein the model codebook is obtained by fusing the feature map and the text features of the description text.

[0049] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0050] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the training method of the large language model based on joint generative pre-training provided by the above methods, the method comprising: obtaining a medical image and a corresponding description text, encoding the medical image through a visual encoder, and obtaining a feature map of the medical image; for each pixel code in the feature map, querying the similar code of the pixel code from the original unified codebook of the multimodal large language model to obtain a codebook map and a quantized feature map corresponding to the codebook map; obtaining a query indication map, and filtering the original unified codebook for similar codes based on the query indication map to obtain a filtered update codebook, wherein the query indication map is determined based on the flattened quantized feature map; attenuating and updating the filtered update codebook through the model codebook of the multimodal large language model to obtain a training update codebook, and training the multimodal large language model based on the training update codebook, wherein the model codebook is obtained by fusing the feature map and the text features of the description text.

[0051] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the processor executes the training method of the large language model based on joint generative pre-training provided by the above methods, the method comprising: obtaining a medical image and a corresponding description text, encoding the medical image through a visual encoder, and obtaining a feature map of the medical image; for each pixel code in the feature map, querying the similar code of the pixel code from the original unified codebook of the multimodal large language model to obtain a codebook map and a quantized feature map corresponding to the codebook map; obtaining a query indication map, and filtering the original unified codebook for similar codes based on the query indication map to obtain a filtered update codebook, wherein the query indication map is determined based on the flattened quantized feature map; attenuating and updating the filtered update codebook through the model codebook of the multimodal large language model to obtain a training update codebook, and training the multimodal large language model based on the training update codebook, wherein the model codebook is obtained by fusing the feature map and the text features of the description text.

[0052] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0053] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0054] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A training method for a large language model based on joint generative pre-training, characterized in that: The method comprises: Acquire a medical image and a corresponding description text, encode the medical image through a visual encoder, and obtain a feature map of the medical image; For each pixel code in the feature map, query similar codes of the pixel code from an original unified codebook of the multimodal large language model to obtain a codebook map and a quantized feature map corresponding to the codebook map; Acquire a query indication graph, and screen the original unified codebook for similar coding based on the query indication graph to obtain a screened updated codebook, wherein the query indication graph is determined based on the flattened quantized feature graph; The screening update codebook is attenuated and updated through the model codebook of the multimodal large language model to obtain a training update codebook, and the multimodal large language model is trained based on the training update codebook, wherein the model codebook is obtained by fusing the feature map and the text features of the description text.

2. The training method of a large language model based on joint generative pre-training according to claim 1, characterized in that: The method of querying similar codes of the pixel codes from the original unified codebook of the multimodal large language model to obtain a codebook graph and a quantized feature graph corresponding to the codebook graph includes: Determining the Euclidean distance between the pixel code and each target code of an original unified codebook of a multimodal large language model; The target code with the smallest Euclidean distance is used as the similarity code corresponding to the pixel code; Combining similar codes corresponding to each pixel code to obtain a codebook graph; The expected value of the codebook image is used as the quantization feature image corresponding to the codebook image.

3. The training method of a large language model based on joint generative pre-training according to claim 1, characterized in that: The obtaining of the query indication graph comprises: Flattening the quantized feature map to obtain a flattened feature map; For each subcode in the original unified codebook, when the subcode exists in the flattened feature map, marking the subcode with a used label; When the sub-code does not exist in the flattened feature map, marking the sub-code with an unused label; A query indication graph is constructed according to the used labels and the unused labels.

4. The training method of a large language model based on joint generative pre-training according to claim 1, characterized in that: The screening of similar codes in the original unified codebook based on the query indication graph to obtain a screened updated codebook includes: Obtaining a first attenuation ratio, and determining a difference between 1 and the first attenuation ratio as a second attenuation ratio; Weighting the original unified codebook by the first attenuation ratio to obtain a first weighted result; The flattened feature graph is screened by the query indication graph to obtain a screened feature graph, and the screened feature graph is weighted by the second attenuation ratio to obtain a second weighted result; The sum of the first weighted result and the second weighted result is used as a filter to update the codebook.

5. The training method of a large language model based on joint generative pre-training according to claim 1, characterized in that: The attenuating and updating the screening update codebook by using the model codebook of the multimodal large language model to obtain the training update codebook includes: weighting the screened updated codebook according to the first attenuation ratio to obtain a third weighted result; weighting the model codebook by a second attenuation ratio to obtain a fourth weighted result; The sum of the third weighted result and the fourth weighted result is used as a training updated codebook.

6. The training method of a large language model based on joint generative pre-training according to claim 2, characterized in that: The method further comprises: Calling a decoder to perform image decoding processing on the codebook image to obtain a reconstructed medical image of the medical image; The reconstructed medical image is encoded by the visual encoder to obtain a reconstructed feature map of the reconstructed medical image, wherein the reconstructed feature map is used to determine a reconstructed codebook map to update the original unified codebook.

7. A training device for a large language model based on joint generative pre-training, characterized in that: The device comprises: An encoding module, used for acquiring a medical image and a corresponding description text, encoding the medical image through a visual encoder, and obtaining a feature map of the medical image; A query module is used to query similar codes of each pixel code in the feature map from an original unified codebook of the multimodal large language model to obtain a codebook map and a quantized feature map corresponding to the codebook map; A screening module, used to obtain a query indication graph, and screen the original unified codebook for similar coding based on the query indication graph to obtain a screened updated codebook, wherein the query indication graph is determined based on the flattened quantized feature graph; A training module is used to attenuate and update the screening update codebook through the model codebook of the multimodal large language model to obtain a training update codebook, and train the multimodal large language model based on the training update codebook, wherein the model codebook is obtained by fusing the feature map and the text features of the description text.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the training method of the large language model based on joint generative pre-training as described in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the training method of a large language model based on joint generative pre-training as described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the training method of a large language model based on joint generative pre-training as described in any one of claims 1 to 6 is implemented.

Images