Multi-modal generative model training method and device and text generation method and device

Abstract

The invention relates to the technical field of artificial intelligence, in particular to a training method and a training device of a multi-modal generation model, and a text generation method and a text generation device. According to the specific implementation scheme, a first training sample is obtained; performing degradation processing on the first training sample to obtain a second training sample; and inputting the first training sample and the second training sample into an initial generation model, and performing preference alignment training on the initial generation model to obtain a multi-modal generation model. According to the method, the high-quality image-text data can be degraded to obtain the non-preference data, the initial generation model is subjected to preference alignment optimization, large model illusion and / or non-rigorous output content are regarded as negative samples in preference alignment optimization training, illusion of a multi-modal generation model is reduced, and the multi-modal generation model can be optimized. And the preciseness of the output content of the multi-modal generation model in the medical scene is effectively improved.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to a training method and a training device for a multimodal generation model, and a text generation method and a generation device. Background Art

[0002] With the rise of technologies such as multi-modal large language models (MLLM), unified processing of data in different modalities and formats has become a current research and application focus. In the process of medical diagnosis and treatment, a large amount of text information and medical image data information will be generated. It is necessary to use a multimodal generation model to generate corresponding text content based on the text and medical image data input by the user, for example, to generate a text description of the B-ultrasound image based on the user's B-ultrasound image and related description of the condition. However, the existing multimodal generation models are designed for general scenarios. Due to the relative scarcity of high-quality image data and text corpora in medical scenarios, it is difficult to support large-scale model training. Therefore, the multimodal generation models trained by existing methods cannot meet the rigorous requirements for generated content in medical scenarios. This leads to the fact that in medical diagnosis and treatment scenarios, multimodal generation models are prone to model hallucinations. For example, when a user inputs an eye color ultrasound image and the corresponding text description, the model finally generates "visible X-ray image, no macular area in the fundus". If model hallucinations occur in medical diagnosis and treatment scenarios, this may lead to serious misjudgment of the patient's condition. Summary of the invention

[0003] The present disclosure provides a multimodal generation model training method, a multimodal generation model training device, a text generation method, a text generation device, an electronic device, and a storage medium.

[0004] According to a first aspect of the present disclosure, a multimodal generative model training method is provided, comprising:

[0005] Obtain a first training sample;

[0006] Performing degradation processing on the first training sample to obtain a second training sample;

[0007] The first training sample and the second training sample are input into an initial generation model, and preference alignment training is performed on the initial generation model to obtain a multimodal generation model.

[0008] According to a second aspect of the present disclosure, there is provided a text generation method, comprising:

[0009] Obtain image data and text data input by the user;

[0010] Inputting the image data and the text data into the multimodal generation model described in any one of the above technical solutions;

[0011] The corresponding reply text is generated by the multimodal generation model.

[0012] According to a third aspect of the present disclosure, a multimodal generative model training device is provided, comprising:

[0013] A first acquisition module is configured to acquire a first training sample;

[0014] A degradation module, configured to perform degradation processing on the first training sample to obtain a second training sample;

[0015] The training module is configured to input the first training sample and the second training sample into an initial generation model, and perform preference alignment training on the initial generation model to obtain a multimodal generation model.

[0016] According to a fourth aspect of the present disclosure, a text generation device includes:

[0017] A second acquisition module is configured to acquire image data and text data input by a user;

[0018] An input module, configured to input the image data and the text data into the multimodal generation model described in any one of the above technical solutions;

[0019] The generation module is configured to generate corresponding reply text through the multimodal generation model.

[0020] According to a fifth aspect of the present disclosure, there is provided an electronic device, including:

[0021] at least one processor; and

[0022] a memory communicatively connected to the at least one processor; wherein,

[0023] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any method in any of the above technical solutions.

[0024] According to a sixth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any one of the methods described in the above technical solutions.

[0025] According to a seventh aspect of the present disclosure, a computer program product is provided, including a computer program, wherein the computer program implements any one of the methods described in the above technical solutions when executed by a processor.

[0026] According to an eighth aspect of the present disclosure, a medical imaging knowledge question and answer system is provided, comprising the multimodal generation model described in any one of the above technical solutions.

[0027] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0029] Figure 1 is a schematic diagram of the steps of the multimodal generative model training method in an embodiment of the present disclosure;

[0030] Figure 2 is a flowchart of a multimodal generative model training method in an embodiment of the present disclosure;

[0031] Figure 3 is a schematic diagram of steps of a text generation method in an embodiment of the present disclosure;

[0032] Figure 4 is a schematic diagram of the structure of a multimodal generation model in an embodiment of the present disclosure;

[0033] Figure 5 is a principle block diagram of a multimodal generative model training device in an embodiment of the present disclosure;

[0034] Figure 6 is a principle block diagram of a text generation device in an embodiment of the present disclosure;

[0035] Figure 7 It is a block diagram of an electronic device used to implement the multimodal generation model training method or text generation method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0036] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0037] In view of the technical problem that in the prior art, multimodal generative models in the field of medical diagnosis and treatment lack high-quality image data and text data of medical scenes to train the models, resulting in the lack of rigorous text content generated by the multimodal generative models and the easy occurrence of model hallucinations, the present disclosure provides a multimodal generative model training method, such as Figure 1 As shown, including:

[0038] Step S101, obtaining a first training sample.

[0039] Specifically, the first training sample in this embodiment refers to high-quality graphic data in the medical field, which may include text data, image data, etc. The reason why there are few high-quality graphic data in the medical field is that, on the one hand, data structured access and centralized management are insufficient. Medical data mainly includes structured data and unstructured data, among which unstructured data (such as text and image data) is difficult to be directly processed by computers, and there are many information systems online in hospitals, lacking unified standards and interface support, resulting in difficulties in data sharing. On the other hand, medical data is not standardized enough. In the construction of medical big data platforms, only limited data collection is often solved, such as data from electronic medical record systems or integrated platform data, while a large amount of physical examination data, Internet consultation records, and historical medical records of external hospitals cannot be obtained, resulting in incomplete data and insufficient volume.

[0040] Step S102: Perform degradation processing on the first training sample to obtain a second training sample.

[0041] Since the acquired high-quality image and text data is limited, the present invention uses a data degrader to degrade the acquired high-quality image and text data, and uses artificially constructed degraded data as "non-preferred" or "click-rejected" samples. A large amount of model training data can be obtained through data degradation processing, which is applied to subsequent preference alignment optimization methods.

[0042] Step S103: input the first training sample and the second training sample into an initial generation model, and perform preference alignment training on the initial generation model to obtain a multimodal generation model.

[0043] The present invention can simultaneously utilize high-quality graphic data in the medical field and non-preferred data obtained by degrading high-quality graphic data to perform preference alignment optimization on the initial generation model. By treating large model hallucinations and or inaccurate output content as negative samples in preference alignment optimization training, the hallucinations of the multimodal generation model are reduced. When high-quality graphic data are limited, the model is trained by expanding negative samples, effectively improving the rigor of the output content of the multimodal generation model in the medical scenario.

[0044] As an optional implementation, step S101, obtaining a first training sample includes:

[0045] A target image and a corresponding target text are obtained as the first training sample.

[0046] Specifically, taking the medical scenario as an example, a large amount of text data and medical image data will be generated in the medical diagnosis and treatment scenario. The target image can be medical image data, including but not limited to X-ray, CT (computed tomography), MRI (magnetic resonance imaging), ultrasound and other image data. These image data can intuitively display the organ structure and pathological conditions in the patient's body, and are an important basis for doctors to make diagnoses. For example, in the diagnosis of lung diseases, doctors can observe the pathological conditions of the lungs by analyzing the patient's CT images, thereby making an accurate diagnosis.

[0047] The target text can include text data such as electronic medical records, patient self-reports, and medical literature. Electronic medical records are the core component of hospital information systems. They contain a large amount of text information such as the patient's basic information, medical history, diagnosis results, treatment plans, etc. Patient self-reports are also one of the important sources of text data. Patients can describe their symptoms, medical history, feelings, etc. to doctors in text or voice. Medical literature is an important resource for medical research and clinical practice. They contain a lot of medical knowledge and experience summaries. By conducting text mining and semantic analysis on medical literature, new medical laws can be discovered, disease development trends can be predicted, and drug efficacy can be evaluated.

[0048] As an optional implementation manner, step S102, performing degradation processing on the first training sample to obtain the second training sample includes:

[0049] Step S102a, performing degradation processing on the target image to obtain a degraded image.

[0050] like Figure 2 As shown, firstly, the first training sample 201 is input into the data degrader 202 to obtain the second training sample 203, and then the first training sample 201 and the second training sample 203 are input into the initial generation model for preference alignment training to obtain the multimodal generation model 204. Wherein. In the degradation process of the first training sample, the data type currently being processed is first determined. If it is image data, the image is degraded to obtain a negative sample of the image data. A degraded image refers to an image sample that does not meet specific conditions, for example, low-quality image data obtained by distorting and deforming a patient's lung CT image.

[0051] Step S102b, performing degradation processing on the target text to obtain a degraded text.

[0052] like Figure 2As shown, during the degradation process of the first training sample, text degradation is performed on the text data, and the key information can be replaced with text that is semantically similar but deviates from the original text, and the low-quality text data obtained after the degradation process is used as the degraded text.

[0053] Degraded images and degraded texts together constitute negative samples. Negative samples can play a vital role in the training process because they help the model learn how to distinguish target categories from non-target categories. By comparing the features of positive and negative samples, the model can learn more accurate and robust classification boundaries.

[0054] As an optional implementation manner, performing degradation processing on the target image to obtain a degraded image includes:

[0055] Degrade the overall structure in the target image.

[0056] In medical image processing, the degree of attention paid to the overall image and key structures often depends on the specific clinical application and diagnostic needs. For example, X-ray images are used to observe the overall morphology of bone structures and certain high-density tissues, such as lungs and hearts. In these images, doctors will pay attention to the morphology, positional relationship, and presence of abnormal structures in the entire anatomical region. For example, MRI (magnetic resonance imaging) images provide richer soft tissue contrast than CT, and can clearly display structures such as the brain, spinal cord, muscles, and blood vessels. In MRI images, doctors will also pay attention to the morphology, positional relationship, and signal intensity changes of the overall structure to assess the health of the tissue. In addition, CT (computed tomography) images can display the three-dimensional structure of the human body through X-ray scanning and reconstruction at multiple levels. When doctors view CT images, they will comprehensively consider information at multiple levels to form an overall understanding of the anatomical region and look for possible lesions or abnormalities. In these application scenarios, it is necessary to pay more attention to the overall structure of the image data, so the overall structure can be degraded to obtain a degraded image.

[0057] As an optional implementation, the degradation processing performed on the overall structure in the target image includes at least one of distortion, deformation, and superimposition of false color processing.

[0058] Distortion refers to the nonlinear transformation of an image, changing the position of pixels in the image, thereby achieving the deformation effect of the image. This processing can be implemented based on mathematical functions or image processing algorithms, such as affine transformation, perspective transformation, elastic transformation, etc. Deformation is similar to distortion to some extent, both involving the change of the position of pixels in the image. However, deformation may focus more on local adjustments to specific areas or key structures in the image to achieve a more refined deformation effect. Deformation can be achieved through a variety of methods, such as grid-based deformation, feature point-based deformation, etc. Pseudocolor Overlay Pseudocolor Overlay is a technique for converting grayscale images or monochrome images into color images. In this process, pixels in the image are mapped to different colors according to certain criteria (such as grayscale value, intensity, etc.), thereby enhancing the visual effect and detail expression of the image. Degradation of the target image in shape or color can be achieved in a variety of ways.

[0059] As an optional implementation manner, performing degradation processing on the target image to obtain a degraded image includes:

[0060] The key structure in the target image is obtained through image segmentation, and the key structure is degraded.

[0061] In some application scenarios, doctors pay more attention to the local key structures of the image rather than the overall structure of the image. For example, ultrasound images are often used to assess the health of key structures such as the fetus, heart, and blood vessels. In these applications, doctors pay special attention to key structures in the image, such as fetal development, heart valve function, and blood flow status of blood vessels. For example, PET (positron emission tomography) and SPECT (single photon emission tomography) images, these nuclear medicine images are mainly used to assess the metabolic activity or blood perfusion of tissues. In these images, doctors pay special attention to the metabolic or perfusion changes in the region of interest (ROI) to determine whether there are tumors, inflammation, or other abnormal metabolic processes. For such images, the key structures can be segmented out first through image segmentation technology, and then the key structures can be degraded.

[0062] As an optional implementation, the key structure in the target image is obtained by image segmentation, and the degradation processing performed on the key structure includes at least one of extraction, erasure, replacement, geometric transformation, and color transformation.

[0063] Extraction refers to the process of separating a specific area in an image (such as the heart, fetus, blood vessels, etc.) from the background in image processing. This usually requires the use of image segmentation techniques, such as color-based segmentation, texture-based segmentation, edge detection-based segmentation, etc., to accurately identify and extract the target area. Erasing refers to removing unnecessary parts or content from an image. Replacement in image processing refers to replacing a key structure in an image (such as the heart) with another structure (such as the lungs). Geometric transformation refers to the transformation of the spatial position or shape of an image to change the position or shape of pixels in the image. Common geometric transformations include translation, rotation, scaling, flipping, affine transformation, perspective transformation, etc. Color transformation refers to the process of adjusting or changing the color information in an image. This can be achieved by adjusting the image's hue, saturation, brightness and other attributes. Combine multiple degradation methods to degrade the target image to obtain degraded images in different situations.

[0064] As an optional implementation, the target text is degraded to obtain a degraded text including:

[0065] The target text and the corresponding rewriting instructions are input into the large language model, and the target text is rewritten according to the rewriting instructions by the large language model, and the corresponding degraded text is output.

[0066] Specifically, the target text input to the large language model can be a text description of the target image, such as "visible fundus color photo", and a rewriting instruction can be input, "Please help me rewrite this sentence. The original meaning can be different, but the content must be roughly the same", and the large language model rewrites it to get "visible X-ray image". The large language model rewrites to get richer degraded text.

[0067] As an optional implementation, rewriting the target text according to the rewriting instruction by using a large language model, and outputting the corresponding degraded text includes:

[0068] Tamper with key information in the target text to obtain a degraded text; and / or

[0069] The target text is expanded to obtain the degraded text.

[0070] Specifically, the first way to rewrite the target text is to tamper with the key information in the target text, for example, rewriting "visible fundus color photo" to "visible X-ray image". The second way is to expand and imagine the image content based on the incomplete image description text. For example, the input image description text "visible image, visible optic cup and optic disc in the fundus" is expanded to "visible image, no macular area in the fundus". By treating large model hallucinations and / or imprecise outputs as negative samples in preference alignment, the rigor of multimodal AI content production in medical scenarios can be effectively improved.

[0071] As an optional implementation, inputting the first training sample and the second training sample into an initial generation model, and performing preference alignment training on the initial generation model to obtain a multimodal generation model includes:

[0072] The first training sample is used as a preferred sample, the second training sample is used as a non-preferred sample, and the difference between the first training sample and the second training sample is calculated according to a preset loss function.

[0073] The initial generative model is trained for preference alignment based on the difference to obtain a multimodal generative model.

[0074] Specifically, the preset loss function for Direct Preference Optimization (DPO) in this embodiment can be expressed as:

[0075]

[0076] Where I represents the input target image; q represents the input target text; Π represents the initial generation model; θ represents the parameters of the initial generation model; r + represents the like sample, i.e. the positive sample; r - Represents the click-rejection sample, that is, the negative sample.

[0077] The meaning of the above formula is to increase the output probability of positive samples and reduce the output probability of negative samples. Through this DPO optimization method, the model's hallucinations can be reduced, the model's inaccurate output can be reduced, and the rigor of the output content of the multimodal generation model in medical scenarios can be improved.

[0078] The present disclosure also adopts a text generation method, such as Figure 3 As shown, including:

[0079] Step S301, obtaining image data and text data input by a user.

[0080] Specifically, the image data may be medical imaging data, such as X-rays, CT images, MRI (magnetic resonance imaging), ultrasound images, etc. The text data may include text data associated with the image data, such as the patient's electronic medical records, patient self-reports, medical literature, and other text data. Figure 2 As shown, image data 205 and text data 206 may be input into a multimodal generation model 204 , and a reply text 207 may be output through the multimodal generation model 204 .

[0081] Step S302: input the image data and the text data into the multimodal generation model in any one of the above embodiments.

[0082] Specifically, Figure 4 As shown in Figure 1, the multimodal generation model can adopt the Transformers structure. The input sequence to the Transformers model contains text tokens and image tokens. The image encoder encodes the input image data into several image feature vectors, and then superimposes Gaussian noise on the vector group according to the diffusion model method. The processed vector group is named M t Next, each vector is input into the Transformers structure in turn, the output result corresponding to the Transformers structure is obtained, and the difference between the group and the real noise vector group at time t-1 is calculated in groups:

[0083]

[0084] Randomly sample noise ε~N(0,I) from a standard Gaussian distribution, consider the model has parameters θ, and output the guessed noise data ε θ (m t ,t,c), calculate the gradient of θ (parameter adjustment signal) according to the following formula:

[0085]

[0086] Qualitatively speaking, the diffusion model is based on m t , time t and generation condition c to guess the random noise added at the current time. The mean square error is calculated between the guessed result and the actual added noise.

[0087] Step S303, generating corresponding reply text through a multimodal generation model.

[0088] For text content, this solution reuses the existing text generation loss function calculation method:

[0089] L text =∑CrossEntropy(T n ,T' n )

[0090] Among them, T n The text feature vector representing the input Transformers structure; T' n Represents the text feature vector output by the Transformers structure.

[0091] The training process of the multimodal generative model disclosed in the present invention may include two parts. One part is based on the generation structure optimization goal, that is, according to Figure 4 The text feature vector T' output by the Transformers structure shown n , image feature vector I' n , generate condition C' nThe two parts optimize the objectives respectively; the other part is DPO optimization, which optimizes the overall feature sequence output by the Transformers structure.

[0092] Specifically, the preset loss function for Direct Preference Optimization (DPO) in this embodiment can be expressed as:

[0093]

[0094] Where I represents the input target image; q represents the input target text; Π represents the initial generation model; θ represents the parameters of the initial generation model; r + represents the like sample, i.e. the positive sample; r - Represents the click-rejection sample, that is, the negative sample.

[0095] The meaning of the above formula is to increase the output probability of positive samples and reduce the output probability of negative samples. Through this DPO optimization method, the model's hallucinations can be reduced, the model's inaccurate output can be reduced, and the rigor of the output content of the multimodal generation model in medical scenarios can be improved.

[0096] During the training process, the multimodal generative model in this embodiment regards large model hallucinations and or inaccurate output content as negative samples in the preference alignment optimization training, thereby reducing the hallucinations of the multimodal generative model. In the case of limited high-quality graphic data, the model is trained by expanding the negative samples, thereby effectively improving the rigor of the output content of the multimodal generative model in medical scenarios.

[0097] The present disclosure also provides a multimodal generative model training device 500, such as Figure 5 As shown, including:

[0098] The first acquisition module 501 is configured to acquire a first training sample.

[0099] Specifically, the first training sample in this embodiment refers to high-quality graphic data in the medical field, which may include text data, image data, etc. The reason why there are few high-quality graphic data in the medical field is that, on the one hand, data structured access and centralized management are insufficient. Medical data mainly includes structured data and unstructured data, among which unstructured data (such as text and image data) is difficult to be directly processed by computers, and there are many information systems online in hospitals, lacking unified standards and interface support, resulting in difficulties in data sharing. On the other hand, medical data is not standardized enough. In the construction of medical big data platforms, only limited data collection is often solved, such as data from electronic medical record systems or integrated platform data, while a large amount of physical examination data, Internet consultation records, and historical medical records of external hospitals cannot be obtained, resulting in incomplete data and insufficient volume.

[0100] The degradation module 502 is configured to perform degradation processing on the first training sample to obtain a second training sample.

[0101] Since the acquired high-quality image and text data is limited, the present invention uses a data degrader to degrade the acquired high-quality image and text data, and uses artificially constructed degraded data as "non-preferred" or "click-rejected" samples. A large amount of model training data can be obtained through data degradation processing, which is applied to subsequent preference alignment optimization methods.

[0102] The training module 503 is configured to input the first training sample and the second training sample into the initial generation model, and perform preference alignment training on the initial generation model to obtain a multimodal generation model.

[0103] The present invention can simultaneously utilize high-quality graphic data in the medical field and non-preferred data obtained by degrading high-quality graphic data to perform preference alignment optimization on the initial generation model. By treating large model hallucinations and or inaccurate output content as negative samples in preference alignment optimization training, the hallucinations of the multimodal generation model are reduced. When high-quality graphic data are limited, the model is trained by expanding negative samples, effectively improving the rigor of the output content of the multimodal generation model in the medical scenario.

[0104] As an optional implementation, the first acquisition module 501 acquires the first training sample including:

[0105] A target image and a corresponding target text are obtained as the first training sample.

[0106] Specifically, taking the medical scenario as an example, a large amount of text data and medical image data will be generated in the medical diagnosis and treatment scenario. The target image can be medical image data, including but not limited to X-ray, CT (computed tomography), MRI (magnetic resonance imaging), ultrasound and other image data. These image data can intuitively display the organ structure and pathological conditions in the patient's body, and are an important basis for doctors to make diagnoses. For example, in the diagnosis of lung diseases, doctors can observe the pathological conditions of the lungs by analyzing the patient's CT images, thereby making an accurate diagnosis.

[0107] The target text can include text data such as electronic medical records, patient self-reports, and medical literature. Electronic medical records are the core component of hospital information systems. They contain a large amount of text information such as the patient's basic information, medical history, diagnosis results, treatment plans, etc. Patient self-reports are also one of the important sources of text data. Patients can describe their symptoms, medical history, feelings, etc. to doctors in text or voice. Medical literature is an important resource for medical research and clinical practice. They contain a lot of medical knowledge and experience summaries. By conducting text mining and semantic analysis on medical literature, new medical laws can be discovered, disease development trends can be predicted, and drug efficacy can be evaluated.

[0108] As an optional implementation, the degradation module 502 includes:

[0109] The image degradation unit is configured to perform degradation processing on the target image to obtain a degraded image.

[0110] like Figure 2 As shown, in the degradation process of the first training sample, the data type currently being processed is first determined. If it is image data, the image is degraded to obtain a negative sample of the image data. A degraded image refers to an image sample that does not meet specific conditions, for example, low-quality image data obtained by distorting and deforming a patient's lung CT image.

[0111] The text degradation unit is configured to perform degradation processing on the target text to obtain a degraded text.

[0112] like Figure 2 As shown, during the degradation process of the first training sample, text degradation is performed on the text data, and the key information can be replaced with text that is semantically similar but deviates from the original text, and the low-quality text data obtained after the degradation process is used as the degraded text.

[0113] Degraded images and degraded texts together constitute negative samples. Negative samples can play a vital role in the training process because they help the model learn how to distinguish target categories from non-target categories. By comparing the features of positive and negative samples, the model can learn more accurate and robust classification boundaries.

[0114] As an optional implementation manner, the image degradation unit performs degradation processing on the target image to obtain a degraded image, including:

[0115] Degrade the overall structure in the target image.

[0116] In medical image processing, the degree of attention paid to the overall image and key structures often depends on the specific clinical application and diagnostic needs. For example, X-ray images are used to observe the overall morphology of bone structures and certain high-density tissues, such as lungs and hearts. In these images, doctors will pay attention to the morphology, positional relationship, and presence of abnormal structures in the entire anatomical region. For example, MRI (magnetic resonance imaging) images provide richer soft tissue contrast than CT, and can clearly display structures such as the brain, spinal cord, muscles, and blood vessels. In MRI images, doctors will also pay attention to the morphology, positional relationship, and signal intensity changes of the overall structure to assess the health of the tissue. In addition, CT (computed tomography) images can display the three-dimensional structure of the human body through X-ray scanning and reconstruction at multiple levels. When doctors view CT images, they will comprehensively consider information at multiple levels to form an overall understanding of the anatomical region and look for possible lesions or abnormalities. In these application scenarios, it is necessary to pay more attention to the overall structure of the image data, so the overall structure can be degraded to obtain a degraded image.

[0117] As an optional implementation manner, the image degradation unit performs degradation processing on the overall structure of the target image including at least one of distortion, deformation, and superimposition of false color processing.

[0118] Distortion refers to the nonlinear transformation of an image, changing the position of pixels in the image, thereby achieving the deformation effect of the image. This processing can be implemented based on mathematical functions or image processing algorithms, such as affine transformation, perspective transformation, elastic transformation, etc. Deformation is similar to distortion to some extent, both involving the change of the position of pixels in the image. However, deformation may focus more on local adjustments to specific areas or key structures in the image to achieve a more refined deformation effect. Deformation can be achieved through a variety of methods, such as grid-based deformation, feature point-based deformation, etc. Pseudocolor Overlay Pseudocolor Overlay is a technique for converting grayscale images or monochrome images into color images. In this process, pixels in the image are mapped to different colors according to certain criteria (such as grayscale value, intensity, etc.), thereby enhancing the visual effect and detail expression of the image. Degradation of the target image in shape or color can be achieved in a variety of ways.

[0119] As an optional implementation manner, the image degradation unit performs degradation processing on the target image to obtain a degraded image, including:

[0120] The key structure in the target image is obtained through image segmentation, and the key structure is degraded.

[0121] In some application scenarios, doctors pay more attention to the local key structures of the image rather than the overall structure of the image. For example, ultrasound images are often used to assess the health of key structures such as the fetus, heart, and blood vessels. In these applications, doctors pay special attention to key structures in the image, such as fetal development, heart valve function, and blood flow status of blood vessels. For example, PET (positron emission tomography) and SPECT (single photon emission tomography) images, these nuclear medicine images are mainly used to assess the metabolic activity or blood perfusion of tissues. In these images, doctors pay special attention to the metabolic or perfusion changes in the region of interest (ROI) to determine whether there are tumors, inflammation, or other abnormal metabolic processes. For such images, the key structures can be segmented out first through image segmentation technology, and then the key structures can be degraded.

[0122] As an optional implementation, the image degradation unit obtains the key structure in the target image by image segmentation, and the degradation processing performed on the key structure includes at least one of extraction, erasure, replacement, geometric transformation, and color transformation.

[0123] Extraction refers to the process of separating a specific area in an image (such as the heart, fetus, blood vessels, etc.) from the background in image processing. This usually requires the use of image segmentation techniques, such as color-based segmentation, texture-based segmentation, edge detection-based segmentation, etc., to accurately identify and extract the target area. Erasing refers to removing unnecessary parts or content from an image. Replacement in image processing refers to replacing a key structure in an image (such as the heart) with another structure (such as the lungs). Geometric transformation refers to the transformation of the spatial position or shape of an image to change the position or shape of pixels in the image. Common geometric transformations include translation, rotation, scaling, flipping, affine transformation, perspective transformation, etc. Color transformation refers to the process of adjusting or changing the color information in an image. This can be achieved by adjusting the image's hue, saturation, brightness and other attributes.

[0124] As an optional implementation manner, the text degradation unit performs degradation processing on the target text to obtain a degraded text including:

[0125] The target text and the corresponding rewriting instructions are input into the large language model, and the target text is rewritten according to the rewriting instructions by the large language model, and the corresponding degraded text is output.

[0126] Specifically, the target text input into the large language model can be a text description of the target image, such as "visible fundus color photograph", and a rewriting instruction can be input, "Please help me rewrite this sentence. The original meaning does not need to be kept, but the content must be roughly the same", and the large language model can rewrite it to obtain "visible X-ray image".

[0127] As an optional implementation, the text degradation unit rewrites the target text according to the rewriting instruction through the large language model, and outputs the corresponding degraded text including:

[0128] Tamper with key information in the target text to obtain a degraded text; and / or

[0129] The target text is expanded to obtain a degraded text.

[0130] Specifically, the first way to rewrite the target text is to tamper with the key information in the target text, for example, rewriting "visible fundus color photo" to "visible X-ray image". The second way is to expand and imagine the image content based on the incomplete image description text. For example, the input image description text "visible image, visible optic cup and optic disc in the fundus" is expanded to "visible image, no macular area in the fundus". By treating large model hallucinations and / or imprecise outputs as negative samples in preference alignment, the rigor of multimodal AI content production in medical scenarios can be effectively improved.

[0131] As an optional implementation, the training module 503 includes:

[0132] The calculation unit is configured to use the first training sample as a preferred sample and the second training sample as a non-preferred sample, and calculate the difference between the first training sample and the second training sample according to a preset loss function.

[0133] The training unit is configured to perform preference alignment training on the initial generation model according to the difference to obtain a multimodal generation model.

[0134] The first training sample is used as a preferred sample, the second training sample is used as a non-preferred sample, and the difference between the first training sample and the second training sample is calculated according to a preset loss function.

[0135] The initial generative model is trained for preference alignment based on the difference to obtain a multimodal generative model.

[0136] Specifically, the preset loss function for DPO optimization in this embodiment can be expressed as:

[0137]

[0138] Where I represents the input target image; q represents the input target text; π represents the initial generation model; θ represents the parameters of the initial generation model; r + represents the like sample, i.e. the positive sample; r - Represents the click-rejection sample, that is, the negative sample.

[0139] The meaning of the above formula is to increase the output probability of positive samples and reduce the output probability of negative samples. Through this DPO optimization method, the model's hallucinations can be reduced, the model's inaccurate output can be reduced, and the rigor of the output content of the multimodal generation model in medical scenarios can be improved.

[0140] The present disclosure also provides a text generation device, such as Figure 6 As shown, including:

[0141] The second acquisition module 601 is configured to acquire image data and text data input by a user.

[0142] Specifically, the image data may be medical imaging data, such as X-rays, CT images, MRI (magnetic resonance imaging), ultrasonic images, etc. The text data may include text data associated with the image data, such as electronic medical records of patients, patient self-reports, medical literature, and other text data.

[0143] The input module 602 is configured to input image data and text data into the multimodal generation model in the above embodiment.

[0144] Specifically, Figure 4 As shown in Figure 1, the multimodal generation model can adopt the Transformers structure. The input sequence to the Transformers model contains text tokens and image tokens. The image encoder encodes the input image data into several image feature vectors, and then superimposes Gaussian noise on the vector group according to the diffusion model method. The processed vector group is named M t Next, each vector is input into the Transformers structure in turn, the output result corresponding to the Transformers structure is obtained, and the difference between the group and the real noise vector group at time t-1 is calculated in groups:

[0145]

[0146] Randomly sample noise ε~N(0,I) from a standard Gaussian distribution, consider the model has parameters θ, and output the guessed noise data ε θ (m t ,t,c), calculate the gradient of θ (parameter adjustment signal) according to the following formula:

[0147]

[0148] Qualitatively speaking, the model is based on m t , time t and generation condition c to guess the random noise added at the current time. The mean square error is calculated between the guessed result and the actual added noise.

[0149] The generation module 603 is configured to generate the corresponding reply text through a multimodal generation model.

[0150] For text content, this solution reuses the existing text generation loss function calculation method:

[0151] L text =∑CrossEntropy(T n ,T' n )

[0152] Among them, T n The text feature vector representing the input Transformers structure; T' n Represents the text feature vector output by the Transformers structure.

[0153] During the training process, the multimodal generative model in this embodiment regards large model hallucinations and or inaccurate output content as negative samples in the preference alignment optimization training, thereby reducing the hallucinations of the multimodal generative model. In the case of limited high-quality graphic data, the model is trained by expanding the negative samples, thereby effectively improving the rigor of the output content of the multimodal generative model in medical scenarios.

[0154] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0155] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0156] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0157] like Figure 7As shown, the device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0158] A number of components in the device 700 are connected to the I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0159] The computing unit 701 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 701 performs the various methods and processes described above, such as a multimodal generation model training method or a text generation method. For example, in some embodiments, the multimodal generation model training method or the text generation method may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the multimodal generation model training method or the text generation method described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to execute the multimodal generation model training method or the text generation method in any other appropriate manner (for example, by means of firmware).

[0160] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0161] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0162] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0163] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0164] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0165] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0166] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of this disclosure can be achieved, and this document is not limited here.

[0167] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A multimodal generative model training method, include: Obtain a first training sample; Performing degradation processing on the first training sample to obtain a second training sample; The first training sample and the second training sample are input into an initial generation model, and preference alignment training is performed on the initial generation model to obtain a multimodal generation model.

2. The method according to claim 1, in, The obtaining of the first training sample comprises: A target image and a corresponding target text are obtained as the first training sample.

3. The method according to claim 2, in, The performing degradation processing on the first training sample to obtain a second training sample comprises: Performing degradation processing on the target image to obtain a degraded image; The target text is degraded to obtain a degraded text.

4. The method according to claim 3, in, The step of performing degradation processing on the target image to obtain a degraded image comprises: The overall structure in the target image is degraded.

5. The method according to claim 4, in, The degradation processing of the overall structure in the target image includes at least one of distortion, deformation, and superimposition of false color processing.

6. The method according to claim 3, in, The step of performing degradation processing on the target image to obtain a degraded image comprises: The key structure in the target image is obtained by image segmentation, and degradation processing is performed on the key structure.

7. The method according to claim 6, in, The key structure in the target image is obtained by image segmentation, and the degradation processing of the key structure includes at least one of extraction, erasure, replacement, geometric transformation, and color transformation.

8. The method according to claim 3, in, The step of performing degradation processing on the target text to obtain the degraded text comprises: The target text and the corresponding rewriting instruction are input into a large language model, the target text is rewritten according to the rewriting instruction by the large language model, and the corresponding degraded text is output.

9. The method according to claim 8, in, The rewriting of the target text according to the rewriting instruction by the large language model and outputting the corresponding degraded text comprises: Falsifying key information in the target text to obtain the degraded text; and / or The target text is expanded to obtain the degraded text.

10. The method according to any one of claims 1 to 9, in, The step of inputting the first training sample and the second training sample into an initial generation model and performing preference alignment training on the initial generation model to obtain a multimodal generation model comprises: Taking the first training sample as a preferred sample and the second training sample as a non-preferred sample, and calculating the difference between the first training sample and the second training sample according to a preset loss function; The initial generation model is subjected to preference alignment training according to the difference to obtain the multimodal generation model.

11. A text generation method, include: Obtain image data and text data input by the user; Inputting the image data and the text data into the multimodal generation model described in any one of claims 1 to 10; The corresponding reply text is generated by the multimodal generation model.

12. A multimodal generative model training device, include: A first acquisition module is configured to acquire a first training sample; A degradation module, configured to perform degradation processing on the first training sample to obtain a second training sample; The training module is configured to input the first training sample and the second training sample into an initial generation model, and perform preference alignment training on the initial generation model to obtain a multimodal generation model.

13. The device according to claim 12, in, The first acquisition module acquires the first training sample including: A target image and a corresponding target text are obtained as the first training sample.

14. The device according to claim 13, in, The degradation module comprises: An image degradation unit, configured to perform degradation processing on the target image to obtain a degraded image; The text degradation unit is configured to perform degradation processing on the target text to obtain a degraded text.

15. The device according to claim 14, in, The image degradation unit performs degradation processing on the target image to obtain a degraded image, including: The overall structure in the target image is degraded.

16. The device according to claim 15, in, The image degradation unit performs degradation processing on the overall structure of the target image, including at least one of distortion, deformation, and superimposition of false color processing.

17. The device according to claim 14, in, The image degradation unit performs degradation processing on the target image to obtain a degraded image, including: The key structure in the target image is obtained by image segmentation, and degradation processing is performed on the key structure.

18. The device according to claim 17, in, The image degradation unit obtains the key structure in the target image by image segmentation, and performs degradation processing on the key structure including at least one of extraction, erasure, replacement, geometric transformation, and color transformation.

19. The device according to claim 14, in, The text degradation unit performs degradation processing on the target text to obtain a degraded text including: The target text and the corresponding rewriting instruction are input into a large language model, the target text is rewritten according to the rewriting instruction by the large language model, and the corresponding degraded text is output.

20. The device according to claim 19, in, The text degradation unit rewrites the target text according to the rewriting instruction by using the large language model, and outputs the corresponding degraded text, including: Falsifying key information in the target text to obtain the degraded text; and / or The target text is expanded to obtain the degraded text.

21. The device according to any one of claims 12 to 20, in, The training module includes: a calculation unit configured to use the first training sample as a preferred sample and the second training sample as a non-preferred sample, and calculate a difference between the first training sample and the second training sample according to a preset loss function; A training unit is configured to perform preference alignment training on the initial generation model according to the difference to obtain the multimodal generation model.

22. A text generation device, include: A second acquisition module is configured to acquire image data and text data input by a user; An input module, configured to input the image data and the text data into the multimodal generation model according to any one of claims 1 to 10; The generation module is configured to generate corresponding reply text through the multimodal generation model.

23. An electronic device, include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 11.

24. A non-transitory computer-readable storage medium storing computer instructions, in, The computer instructions are used to cause the computer to execute the method according to any one of claims 1-11.

25. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 11.

26. A medical imaging knowledge question-answering system, comprising the multimodal generation model described in any one of claims 1-10.

Images