Training method and device based on joint generative pre-training large language model

Abstract

The application provides a training method and device based on a joint generative pre-training large language model, acquires a medical image and a description text, encodes through a visual encoder to obtain a feature map; for each pixel code in the feature map, queries a similar code of the pixel code from an original unified codebook of a multi-modal large language model to obtain a codebook map and a quantized feature map; acquires a query indication map, performs similar code filtering on the original unified codebook to obtain a filtered updated codebook. Finally, the model codebook of the multi-modal large language model is updated in a decay manner to obtain a training updated codebook, and the multi-modal large language model is trained based on the training updated codebook. Through the application, the mapping learning from visual information to text information in the medical field in the prior art becomes complex and difficult, and the information integration is not accurate enough, which limits the applicability and application effect of the multi-modal large language model in the medical field.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical treatment, and in particular to a training method and device of a large language model based on joint generative pre-training. BACKGROUND

[0002] In the medical field, especially in the process of surgery and diagnosis, it is of great application value to automatically analyze medical images and generate related text descriptions. This automation technology not only can significantly reduce the workload of doctors, but also can improve the accuracy and efficiency of surgery and diagnosis. Traditional methods usually rely on single modal (such as pure vision or pure text) data processing, which often shows its limitations in dealing with complex medical data.

[0003] At present, multi-modal learning, that is, using both visual and textual data, has become a trend. However, the multi-modal methods in the prior art still face some challenges in medical applications. First, there are significant differences in characteristics between medical images and related text data, which makes it complex and difficult to learn the mapping from vision to text directly. Second, although multi-modal methods improve the knowledge fusion between modalities by jointly optimizing the visual encoder and the text decoder, they usually fail to fully utilize the intrinsic relationship between vision and text, resulting in insufficient and inaccurate integration of information. In addition, multi-modal learning methods often rely on pre-trained visual encoders and text decoders, which are usually trained on non-medical data, so their applicability and effectiveness in medical scenarios are limited. SUMMARY

[0004] The present application provides a training method and device of a large language model based on joint generative pre-training, to solve the defects that the mapping learning from visual information to text information of the multi-modal large language model in the medical field in the prior art becomes complex and difficult, and the information integration is not accurate enough, resulting in the limitation of the applicability and application effect of the multi-modal large language model in the medical field.

[0005] The present application provides a training method of a large language model based on joint generative pre-training, the method comprising the following steps:

[0006] Obtaining a medical image and a corresponding description text, encoding the medical image through a visual encoder to obtain a feature map of the medical image;

[0007] For each pixel code in the feature map, querying a similar code of the pixel code from an original unified codebook of a multi-modal large language model to obtain a codebook map and a quantized feature map corresponding to the codebook map;

[0008] obtain a query indication graph based on the flattened quantization feature map, and perform screening of similar encodings of the original unified codebook based on the query indication graph to obtain a screened updated codebook;

[0009] perform attenuated updating of the screened updated codebook by using a model codebook of the multi-modal large language model to obtain a training updated codebook, and train the multi-modal large language model based on the training updated codebook, wherein the model codebook is obtained by fusing the feature map and the text features of the description text.

[0010] In some embodiments, the querying of the similar encodings of the pixel encodings from the original unified codebook of the multi-modal large language model to obtain a codebook graph and a quantization feature map corresponding to the codebook graph comprises:

[0011] determining the Euclidean distance between the pixel encoding and each target encoding of the original unified codebook of the multi-modal large language model;

[0012] taking the target encoding with the smallest Euclidean distance as the similar encoding corresponding to the pixel encoding;

[0013] combining the similar encoding corresponding to each pixel encoding to obtain a codebook graph;

[0014] taking the expected value of the codebook graph as the quantization feature map corresponding to the codebook graph.

[0015] In some embodiments, the obtaining of the query indication graph comprises:

[0016] performing flattening processing on the quantization feature map to obtain a flattened feature map;

[0017] for each sub-encoding in the original unified codebook, marking the sub-encoding with a used label when the sub-encoding exists in the flattened feature map;

[0018] marking the sub-encoding with an unused label when the sub-encoding does not exist in the flattened feature map;

[0019] constructing a query indication graph based on the used label and the unused label.

[0020] In some embodiments, the screening of similar encodings of the original unified codebook based on the query indication graph to obtain a screened updated codebook comprises:

[0021] obtaining a first attenuation ratio and determining the difference between 1 and the first attenuation ratio as a second attenuation ratio;

[0022] performing weighting on the original unified codebook by using the first attenuation ratio to obtain a first weighted result;

[0023] The query indication map is used to screen the flattened feature map, to obtain a screened feature map, and the second attenuation ratio is used to weight the screened feature map, to obtain a second weighted result;

[0024] The sum of the first weighted result and the second weighted result is taken as a screened updated codebook.

[0025] In some embodiments, the model codebook of the multi-modal large language model is used to perform attenuation updating on the screened updated codebook, to obtain a training updated codebook, including:

[0026] The screened updated codebook is weighted according to a first attenuation ratio, to obtain a third weighted result;

[0027] The model codebook is weighted according to a second attenuation ratio, to obtain a fourth weighted result;

[0028] The sum of the third weighted result and the fourth weighted result is taken as a training updated codebook.

[0029] In some embodiments, the method further includes:

[0030] A decoder is called to perform image decoding processing on the codebook map, to obtain a reshaped medical image of the medical image;

[0031] The reshaped medical image is encoded by the visual encoder, to obtain a reshaped feature map of the reshaped medical image, wherein the reshaped feature map is used to determine a reshaped codebook map, and the original unified codebook is updated.

[0032] The application also provides a training device based on a joint generative pre-training large language model, which includes the following modules:

[0033] An encoding module is configured to acquire a medical image and corresponding description text, and encode the medical image by a visual encoder to obtain a feature map of the medical image;

[0034] A query module is configured to query, for each pixel code in the feature map, a similar code of the pixel code from an original unified codebook of a multi-modal large language model, to obtain a codebook map and a corresponding quantized feature map of the codebook map;

[0035] A screening module is configured to acquire a query indication map, and perform screening of similar codes of the original unified codebook based on the query indication map, to obtain a screened updated codebook, wherein the query indication map is determined based on a flattened quantized feature map;

[0036] The training module is configured to perform attenuated updating on the screening update codebook by using a model codebook of the multi-modal large language model to obtain a training update codebook, and train the multi-modal large language model based on the training update codebook, wherein the model codebook is obtained by fusing the feature map and the text features of the description text.

[0037] The present application also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the training method of the jointly pre-trained large language model based on generative pre-training according to any one of the above when executing the computer program.

[0038] The present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program is executable by a processor to implement the training method of the jointly pre-trained large language model based on generative pre-training according to any one of the above.

[0039] The present application also provides a computer program product comprising a computer program, wherein the computer program is executable by a processor to implement the training method of the jointly pre-trained large language model based on generative pre-training according to any one of the above.

[0040] The present application provides a training method and device of a large language model based on jointly pre-trained generative pre-training,

[0041] By designing an efficient visual encoder and a unified codebook strategy, the feature representations of visual images and text data can be optimized, the deep integration of cross-modal knowledge can be achieved, the problems of cooperation of visual and text modal knowledge and accurate fusion of multi-modal knowledge in the medical field are solved, and the large language model can naturally and comprehensively understand and integrate medical images and text data, thereby improving the applicability and application effect of the large language model in the medical application field. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0043] Figure 1 is a flowchart of the training method of the large language model based on jointly pre-trained generative pre-training provided by the present application.

[0044] Figure 2 is a principle diagram of the training method of the large language model based on jointly pre-trained generative pre-training provided by the present application.

[0045] Figure 3 is a structural schematic diagram of the training device for the large language model based on joint generative pre-training provided by the present application.

[0046] Figure 4 is a physical structure schematic diagram of an electronic device provided by the present application. DETAILED DESCRIPTION

[0047] To make the objectives, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below with reference to the drawings in the present application. Obviously, the described embodiments are some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the protection scope of the present application.

[0048] The training method for the large language model based on joint generative pre-training of the present application will be described below. Figures 1-2 Figure 1 is a flow schematic diagram of the training method for the large language model based on joint generative pre-training provided by the present application, as shown in Figure 1 , the method comprises the following steps 101 to 104.

[0049] Step 101, acquire a medical image and a corresponding description text, encode the medical image through a visual encoder to obtain a feature map of the medical image.

[0050] Firstly, a large number of medical images I can be collected when performing pathological analysis and clinical, and these medical images are generally used to display the pathological condition or clinical features of a patient, and a corresponding description text T needs to be collected for each medical image, which is generally used to describe the medical image, that is, to describe the pathological condition or clinical features of the patient.

[0051] Next, feature extraction is performed on the medical image and the description text. Referring to Figure 2 , Figure 2 is a principle schematic diagram of the training method for the large language model based on joint generative pre-training provided by the present application, as shown in Figure 2 , the medical image I is encoded through a visual encoder E to obtain a feature map of the medical image, and the description text T can be encoded through a text encoder to obtain a text feature of the description text. The visual encoder and the text encoder can use a visual text contrast pre-training model (CLIP), or other image encoders or text encoders such as a Transformer model, which are not limited here.

[0052] ​Because the feature maps of medical images and the feature dimensions of text features are different, they cannot be directly fed into a multimodal large language model for prediction. To enable the multimodal large language model to learn the fused features of vision and text, this embodiment of the invention designs a unified codebook C, which can be viewed as a pre-built temporary feature database. This feature database is continuously updated by extracting feature maps and text features from medical images and storing them in it. This original unified codebook is pre-built based on the fusion of feature maps from historical medical images and the corresponding descriptive text features. The fusion process can involve sequentially scanning the feature maps to form a one-dimensional image sequence feature, which is then merged with the text features.

[0053] Step 102: For each pixel encoding in the feature map, query the similar encodings of the pixel encodings from the original unified codebook of the multimodal large language model to obtain the codebook map and the corresponding quantized feature map.

[0054] Here, regarding feature maps For each pixel code z in the multimodal large language model, the similar code Q(z; C) of pixel code z is queried from the original unified codebook C of the multimodal large language model to obtain the codebook image. and the quantization feature map corresponding to the codebook image. The specific process is as follows: determine the Euclidean distance e between the pixel code z and each target code in the original unified codebook C of the multimodal large language model, and then take the target code with the smallest Euclidean distance as the similarity code e(k) corresponding to the pixel code z. The calculation formula is expressed as the following formula (1):

[0055] (1)

[0056] In the above formula (1), z represents the feature map. Each pixel in the code is encoded, and K represents the total number of target codes in the original Unicode C. This represents the k-th target encoding. This indicates the calculation of Euclidean distance.

[0057] After determining the similarity code corresponding to each pixel code z, the similarity code corresponding to each pixel code z is... The codebook diagram is obtained by combining the codebooks. ,Right now Where i and j represent the pixel encoding z corresponding to the feature map The coordinates in the codebook image are then used. Expected value E ( () as the quantization feature map corresponding to the codebook image .

[0058] Step 103, obtaining a query indication map, and screening the original unified codebook based on the query indication map to obtain a screened updated codebook.

[0059] Firstly, a query indication map MI is obtained. The process of constructing the query indication map is as follows: firstly, the quantized feature map Z is flattened to obtain a flattened feature map Z. The quantized feature map Z is two-dimensional, and is mapped to one dimension through the flattening process, which is convenient for alignment with the original unified codebook. The query indication map MI is used to record whether each code in the original unified codebook C exists in the flattened feature map Z. Therefore, after obtaining the flattened feature map, for each sub-code in the original unified codebook C, if the sub-code exists in the flattened feature map Z, it means that the sub-code has been used, and a used label is marked for the sub-code, for example, marked as 1. If the sub-code does not exist in the flattened feature map, it means that the sub-code has not been used, and an unused label is marked for the sub-code, for example, marked as 0. Thus, the used codes can be screened from the original unified codebook according to the query indication map MI.

[0060] Next, the screening of similar codes of the original unified codebook is performed based on the query indication map to obtain a screened updated codebook. In an embodiment of the present application, the screening process can be realized by an exponential moving average method combined with a preset first decay ratio . Specifically, the first decay ratio is obtained, and the difference between 1 and the first decay ratio is determined as a second decay ratio . The original unified codebook C is weighted by the first decay ratio to obtain a first weighted result. The flattened feature map Z is screened by the query indication map MI to obtain a screened feature map (MI*Z). The screening process is also the process of feature vector multiplication calculation between the feature map and the codebook. The screened feature map MI*Z is weighted by the second decay ratio to obtain a second weighted result. Finally, the sum of the first weighted result and the second weighted result is taken as the screened updated codebook , which is represented by formula (2):

[0061] (2)

[0062] In the above formula (2), MI represents the query indication map, Z represents the flattened feature map, represents the original unified codebook, represents the screened updated codebook, * represents the feature vector multiplication calculation, and represents the first decay ratio. ​​

[0063] Step 104, attenuating updating the screening update codebook through the model codebook of the multi-modal large language model to obtain a training update codebook, and training the multi-modal large language model based on the training update codebook.

[0064] In order to ensure that the feature map of the medical image and the text feature are close to each other, and facilitate the fusion of the visual feature and the text feature. Therefore, the embodiment of the present application periodically updates the original unified codebook through the feature map and the text feature. The original unified codebook here is also the screening update codebook obtained by screening the query indication map, and is therefore also denoted as C. The updating method is to update the model codebook of the multi-modal large language model The screening update codebook C is attenuated to obtain an updated training update codebook , wherein the model codebook of the multi-modal large language model is obtained by fusing the feature map Z and the text feature of the description text T, and the attenuation updating method is still the exponential moving average method.

[0065] Specifically, according to a first attenuation ratio The screening update codebook C is weighted to obtain a third weighted result; and a second attenuation ratio is used to weight the model codebook to obtain a fourth weighted result; and the sum of the third weighted result and the fourth weighted result is taken as the training update codebook , which is denoted as formula (3) as follows:

[0066]

[0067] In the above formula (3), denotes the screening update codebook, denotes the training update codebook obtained after updating, denotes the model codebook of the multi-modal large language model, and denotes the first attenuation ratio.

[0068] Finally, the multi-modal large language model is trained based on the training update codebook , and in the actual training process, the updating process of the training update codebook is also the training process of the multi-modal large language model.

[0069] In subsequent training, the updated training codebook is used as the original unified codebook for the next training round. Steps 101 to 104 are executed again using the medical images and corresponding descriptive text from the next round to update the original unified codebook. This continuously enables the large language model to effectively learn visual feature maps and text features. The updated original unified codebook allows for the alignment of visual and text features, enabling the multimodal large language model to perceive and more effectively fuse and understand visual and text features. This, in turn, allows for the automatic parsing and generation of accurate text descriptions or text responses for medical decision-making.

[0070] In some embodiments, such as Figure 2 As shown, during the process of updating the original unified encoding and training the multimodal large language model, the decoder (such as...) can also be called. Figure 2 The VQ-GAN decoder in the codebook graph Image decoding is performed to obtain a reconstructed medical image of medical image I. Feature decoding can serve as a feedback process for feature extraction or feature fusion, based on a codebook composed of similar codes to the medical image. The features can be decoded to reconstruct an image. The reconstructed medical image can then be compared with the original medical image I. For example, loss functions can be constructed and fed back into the visual encoder and the multimodal large language model to verify the effectiveness of the updates to the visual encoder and the original unified codebook.

[0071] In some embodiments, the reconstructed medical image can also be accompanied by corresponding text descriptions, which can then be used as training data for the next round. Specifically, the reconstructed medical image is encoded using a visual encoder to obtain a reconstructed feature map. This reconstructed feature map can be used to determine the reconstructed codebook image. That is, similar codes corresponding to the reconstructed feature map are queried from the original unified codebook C of the multimodal large language model to obtain the codebook image and the quantized reconstructed feature map corresponding to the reconstructed codebook image. Then, similar codes are filtered, and finally the original unified codebook C is updated, thereby helping the multimodal large language model training. The update process can refer to steps 103 to 104, which will not be repeated here.

[0072] The resulting multimodal large language model, after training, can be applied to computer-assisted surgical systems or medical systems for any type of lesion and surgical procedure, including but not limited to highly variable surgical types, to meet diverse clinical requirements. By acquiring medical images related to surgical pathology and corresponding image description text in real time, and providing timely text responses, it can help output medical decisions and provide doctors with more accurate diagnostic and treatment support.

[0073] The training method of the large language model based on joint generative pre-training provided by the application can realize in-depth fusion of cross-modal knowledge by designing an efficient visual encoder and a unified codebook strategy, and solve the problems of cooperation of visual and text modal knowledge and accurate fusion of multi-modal knowledge in the medical field. In this way, the multi-modal large language model can naturally and comprehensively understand and fuse medical images and text data. Furthermore, the multi-modal large language model is trained by using medical images and corresponding text as training data, thereby improving the applicability and application effect of the model in the medical application field.

[0074] The training device of the large language model based on joint generative pre-training provided by the application is described below. The training device of the large language model based on joint generative pre-training described below can be correspondingly referred to the training method of the large language model based on joint generative pre-training described above.

[0075] Referring to Figure 3 , Figure 3 is a structural schematic diagram of the training device of the large language model based on joint generative pre-training provided by the application, as Figure 3 shown, the training device of the large language model based on joint generative pre-training includes an encoding module 301, a query module 302, a screening module 303, and a training module 304. The encoding module 301 is configured to obtain a medical image and corresponding description text, encode the medical image by a visual encoder, and obtain a feature map of the medical image. The query module 302 is configured to query a similar code of each pixel code in the feature map from an original unified codebook of a multi-modal large language model, obtain a codebook map and a corresponding quantized feature map of the codebook map, and determine the query instruction map based on the flattened quantized feature map. The screening module 303 is configured to obtain a query instruction map, screen similar codes of the original unified codebook based on the query instruction map, and obtain a screening updated codebook. The training module 304 is configured to perform attenuation update on the screening updated codebook by a model codebook of the multi-modal large language model, obtain a training updated codebook, and train the multi-modal large language model based on the training updated codebook. The model codebook is obtained by fusing the feature map and the text features of the description text.

[0076] Figure 4 is a structural schematic diagram of an electronic device provided by the application, as Figure 4As shown, the electronic device can include a processor 410, a communications interface 420, a memory 430, and a communications bus 440, wherein the processor 410, the communications interface 420, and the memory 430 communicate with each other through the communications bus 440. The processor 410 can invoke the logic instructions in the memory 430 to execute a training method based on a jointly generated pre-training large language model, which includes: obtaining a medical image and a corresponding description text, encoding the medical image through a visual encoder to obtain a feature map of the medical image; for each pixel code in the feature map, querying a similar code of the pixel code from an original unified codebook of a multi-modal large language model to obtain a codebook graph and a corresponding quantized feature map of the codebook graph; obtaining a query indication graph and screening similar codes of the original unified codebook based on the query indication graph to obtain a screened and updated codebook, wherein the query indication graph is determined based on the flattened quantized feature map; performing decay update on the screened and updated codebook through a model codebook of the multi-modal large language model to obtain a training updated codebook, and training the multi-modal large language model based on the training updated codebook, wherein the model codebook is obtained by fusing the feature map and a text feature of the description text.

[0077] In addition, the logic instructions in the memory 830 described above can be implemented in the form of a software functional unit and sold or used as an independent product, which can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various program code storage media.

[0078] In another aspect, the present application also provides a computer program product comprising a computer program, which can be stored on a non-transitory computer-readable storage medium, and the computer program can be executed by a processor to enable a computer to perform the training method of the jointly generative pre-training based large language model provided by the above method, which comprises: obtaining a medical image and corresponding description text, encoding the medical image by a visual encoder to obtain a feature map of the medical image; for each pixel code in the feature map, querying a similar code of the pixel code from an original unified codebook of a multi-modal large language model to obtain a codebook graph and a corresponding quantized feature map of the codebook graph; obtaining a query indication graph and screening similar codes of the original unified codebook based on the query indication graph to obtain a screened and updated codebook, wherein the query indication graph is determined based on the flattened quantized feature map; performing decay updating on the screened and updated codebook by a model codebook of the multi-modal large language model to obtain a training updated codebook, and training the multi-modal large language model based on the training updated codebook, wherein the model codebook is obtained by fusing the feature map and a text feature of the description text.

[0079] In yet another aspect, the present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the training method of the jointly generative pre-training based large language model provided by the above method, which comprises: obtaining a medical image and corresponding description text, encoding the medical image by a visual encoder to obtain a feature map of the medical image; for each pixel code in the feature map, querying a similar code of the pixel code from an original unified codebook of a multi-modal large language model to obtain a codebook graph and a corresponding quantized feature map of the codebook graph; obtaining a query indication graph and screening similar codes of the original unified codebook based on the query indication graph to obtain a screened and updated codebook, wherein the query indication graph is determined based on the flattened quantized feature map; performing decay updating on the screened and updated codebook by a model codebook of the multi-modal large language model to obtain a training updated codebook, and training the multi-modal large language model based on the training updated codebook, wherein the model codebook is obtained by fusing the feature map and a text feature of the description text.

[0080] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed to multiple network units. Part or all of the modules can be selected to achieve the purposes of the embodiments according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0081] Through the description of the above embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software and the necessary general hardware platform, and of course can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0082] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A training method based on a joint generative pre-training large language model, characterized in that, The method includes: Acquire medical images and corresponding descriptive text, and encode the medical images using a visual encoder to obtain feature maps of the medical images; For each pixel encoding in the feature map, similar encodings are queried from the original unified codebook of the multimodal large language model to obtain the codebook map and the quantized feature map corresponding to the codebook map; Obtain a query indicator map, and filter the original unified codebook for similar codes based on the query indicator map to obtain a filtered updated codebook, wherein the query indicator map is determined based on the flattened quantization feature map; The filtered and updated codebook is decayed and updated using the model codebook of the multimodal large language model to obtain a training updated codebook, and the multimodal large language model is trained based on the training updated codebook. The model codebook is obtained by fusing the feature map and the text features of the description text.

2. The training method of a large language model based on joint generative pre-training according to claim 1, characterized in that, The step of querying similar codes for the pixel codes from the original unified codebook of the multimodal large language model to obtain the codebook image and the corresponding quantized feature map includes: Determine the Euclidean distance between the pixel encoding and each target encoding in the original unified codebook of the multimodal large language model; The target code with the smallest Euclidean distance is used as the similarity code corresponding to the pixel code; The similar codes corresponding to each pixel code are combined to obtain the codebook image; The expected value of the codebook graph is used as the quantization feature map corresponding to the codebook graph.

3. The training method of a joint generation-based pre-training large language model according to claim 1, wherein, The obtained query indicator diagram includes: The quantized feature map is flattened to obtain a flattened feature map; For each sub-encoder in the original unified codebook, if the sub-encoder exists in the flattened feature map, a label has been used to mark the sub-encoder; When the sub-encoder does not exist in the flattened feature map, the sub-encoder is marked as unused. A query indication graph is constructed based on the used tags and the unused tags.

4. The training method for a large language model based on joint generative pre-training according to claim 1, characterized in that, The step of filtering the original unified codebook based on the query indicator graph to obtain the filtered updated codebook includes: Obtain the first attenuation ratio, and determine the difference between 1 and the first attenuation ratio as the second attenuation ratio; The original unified codebook is weighted using the first attenuation ratio to obtain a first weighted result. The flattened feature map is filtered by the query indicator map to obtain a filtered feature map, and the filtered feature map is weighted by the second attenuation ratio to obtain a second weighted result. The sum of the first weighted result and the second weighted result is used as the filter and update codebook.

5. The training method for a large language model based on joint generative pre-training according to claim 1, characterized in that, The step of decaying and updating the selected update codebook using the model codebook of the multimodal large language model to obtain the training update codebook includes: The filtered updated codebook is weighted according to the first decay ratio to obtain the third weighted result; The model codebook is weighted using a second attenuation ratio to obtain a fourth weighted result; The sum of the third weighted result and the fourth weighted result is used as the training update codebook.

6. The training method for a large language model based on joint generative pre-training according to claim 2, characterized in that, The method further includes: The decoder is invoked to perform image decoding processing on the codebook image to obtain the reconstructed medical image of the medical image; The reconstructed medical image is encoded by the visual encoder to obtain a reconstructed feature map of the reconstructed medical image, wherein the reconstructed feature map is used to determine the reconstructed codebook map and update the original unified codebook.

7. A training device for a large language model based on joint generative pre-training, characterized in that, The device includes: The encoding module is used to acquire medical images and corresponding descriptive text, and to encode the medical images using a visual encoder to obtain feature maps of the medical images. The query module is used to query similar codes for each pixel code in the feature map from the original unified codebook of the multimodal large language model, so as to obtain the codebook map and the quantized feature map corresponding to the codebook map. A filtering module is used to obtain a query indicator map and filter the original unified codebook based on the query indicator map to obtain a filtered updated codebook, wherein the query indicator map is determined based on the flattened quantization feature map; The training module is used to perform decay updates on the filtered update codebook using the model codebook of the multimodal large language model to obtain a training update codebook, and to train the multimodal large language model based on the training update codebook, wherein the model codebook is obtained by fusing the feature map and the text features of the description text.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the training method for a large language model based on joint generative pre-training as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the training method for a large language model based on joint generative pre-training as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the training method for a large language model based on joint generative pre-training as described in any one of claims 1 to 6.

Images