A method and apparatus for optimizing direct preferences in CT reports based on visual attention masks.

Abstract

This application discloses a method and apparatus for direct preference optimization (DPO) of CT reports based on visual attention masking, relating to the field of medical image processing technology. The method includes: actively blocking access to visual information through a visual attention masking mechanism, causing the model to generate hallucinatory outputs based on its own linguistic prior knowledge while visual access is blocked, thus forming non-preferred reports. This effectively suppresses false positive and false negative hallucinations. Furthermore, a reference-guided generation strategy ensures a high degree of consistency in syntactic structure between preferred and non-preferred reports, allowing DPO training to focus on distinguishing factual content from hallucinatory descriptions, avoiding shortcut learning, and significantly improving hallucination suppression efficiency. This solves the problems of existing technologies that over-rely on linguistic prior knowledge leading to factual hallucinations, lack of mechanisms to construct non-preferred samples by combining visual evidence, and inability to improve clinical accuracy of the model without requiring a large amount of additional labeled data.

Description

Technical Field

[0001] This application relates to the field of medical image processing technology, and in particular to a method and apparatus for direct preference optimization of CT reports based on visual attention masking. Background Technology

[0002] Computed tomography (CT) medical image reporting plays a crucial role in clinical decision-making, transforming complex image data into structured diagnostic evidence. However, the current manual reporting model heavily relies on the expertise of radiologists, and the high workload and complexity of CT interpretation increase the risk of misdiagnosis and missed diagnosis. This has spurred increasing attention to automated report generation technologies, but progress in the field of CT remains significantly limited compared to X-ray imaging due to the volumetric complexity of three-dimensional scans and the scarcity of labeled data.

[0003] In recent years, multimodal large language models (MLLMs) have shown potential in advancing this field. However, these methods often rely excessively on the prior linguistic knowledge of large language models (LLMs), leading to "illusion" phenomena, which manifest as misinterpretations of image features or the generation of descriptions that do not conform to the facts of the images, resulting in false positive and false negative results, ultimately compromising the reliability of diagnosis.

[0004] To alleviate such illusions, existing techniques employ reinforcement learning based on human feedback, but this faces significant challenges in the medical field, such as the difficulty of training reward models and the high complexity of data labeling. Therefore, Direct Preference Optimization (DPO) is gaining increasing attention as a more practical alternative.

[0005] Despite some progress, existing techniques still have significant limitations. For example, in constructing preference pairs, current techniques primarily build non-preference samples at the text level, failing to explicitly incorporate visual evidence. This fails to address the fundamental problem that MLLM tends to ignore visual evidence and rely on linguistic priors. Furthermore, the lack of explicit optimization strategies for syntactic consistency between preference and non-preference reports introduces structural differences, potentially leading to shortcut learning and ultimately weakening the hallucination suppression effect. Additionally, these methods overly rely on large-scale annotated radiology datasets. Considering the high cost and low availability of CT annotation data, this further limits the scale of preference data and the overall performance of the model.

[0006] In summary, existing CT report generation methods based on large language models rely excessively on prior linguistic knowledge, leading to factual illusions. Furthermore, existing preference optimization methods lack mechanisms to construct non-preference samples by incorporating visual evidence, thus failing to improve the clinical accuracy of the model without requiring a large amount of additional labeled data. These issues urgently need to be addressed. Summary of the Invention

[0007] This application provides a method and apparatus for direct preference optimization of CT reports based on visual attention masking, in order to solve the problems of existing technologies that rely too much on prior linguistic knowledge, leading to factual illusions, and lack a mechanism to construct non-preference samples by combining visual evidence, thus failing to improve the clinical accuracy of the model without a large amount of additional labeled data.

[0008] The first aspect of this application provides a method for direct preference optimization of CT reports based on visual attention masking, comprising the following steps: acquiring multiple types of three-dimensional CT image data, fine-tuning a pre-constructed basic model using pre-processed data of the multiple types of three-dimensional CT image data to generate a corresponding basic fine-tuning model, encoding two-dimensional CT image slices in the pre-processed data into multiple embedding vectors, and performing random masking sampling on the multiple embedding vectors to obtain a masked CT image; performing preset sentence splitting and truncation operations on a preset actual preference report to generate a truncation prefix for each independent sentence, and then optimizing the masked CT image. The truncated prefix of the current independent sentence is input into the basic fine-tuning model to generate a new sentence suffix. Based on the new sentence suffix and the truncated prefix, a preset sentence concatenation and combination operation is performed to generate an initial non-preference sample that meets preset consistency requirements. A preset semantic similarity filtering operation is performed on the initial non-preference sample to obtain the corresponding target non-preference report. Based on the two-dimensional CT image slice, the actual preference report, and the target non-preference report, and combined with a preset composite loss function, a preset direct preference optimization operation is performed on the basic fine-tuning model to generate a target CT report that meets preset quality requirements.

[0009] Optionally, in one embodiment of this application, the step of acquiring multiple 3D CT image data and fine-tuning a pre-constructed basic model using preprocessed data of the multiple 3D CT image data to generate a corresponding basic fine-tuning model includes: unifying the resolution and pixel spacing parameters of the multiple 3D CT image data to obtain corresponding parameterized CT image data, and using a preset cubic spline interpolation strategy to extract multiple 2D CT image slices from the parameterized CT image data at target intervals; standardizing the format and character encoding of the real report text corresponding to the multiple 2D CT image slices to obtain corresponding standard report text; determining the preprocessed data based on the multiple 2D CT image slices and the standard report text, and performing a preset supervised fine-tuning operation on the basic model using the preprocessed data to obtain the basic fine-tuning model; encoding the 2D CT image slices in the preprocessed data into multiple embedding vectors using the visual encoder of the basic fine-tuning model, and performing random mask sampling on the multiple embedding vectors based on a preset mask threshold to obtain a masked CT image.

[0010] Optionally, in one embodiment of this application, the step of performing a preset sentence splitting and truncation operation on the preset actual preference report to generate a truncated prefix for each independent sentence, and inputting the masked CT image and the truncated prefix of the current independent sentence into the basic fine-tuning model to generate a new sentence suffix, and performing a preset sentence concatenation and combination operation based on the new sentence suffix and the truncated prefix to generate an initial non-preference sample that meets the preset consistency requirements, includes: performing a preset sentence-level splitting operation on the actual preference report to generate multiple independent sentences, and determining the... The truncation position is used to split the corresponding independent sentence into a truncated prefix and an actual suffix according to the truncation position; the masked CT image, the truncated prefix of the current independent sentence, and the generated answers of all independent sentences before the current independent sentence are input into the basic fine-tuning model to generate a new independent sentence suffix; the truncated prefix of the current independent sentence and the new independent sentence suffix are concatenated to obtain the concatenated independent sentence of the current independent sentence, and the concatenated independent sentences of each independent sentence are combined to construct an initial non-preference sample that meets the preset consistency requirement with the actual preference report.

[0011] Optionally, in one embodiment of this application, the step of performing a preset semantic similarity filtering operation on the initial non-preference sample to obtain a corresponding target non-preference report, and performing a preset direct preference optimization operation on the basic fine-tuning model based on the two-dimensional CT image slice, the actual preference report, and the target non-preference report, and in conjunction with a preset composite loss function, to generate a target CT report that meets preset quality requirements, includes: calculating the cosine similarity between the initial non-preference sample and the actual preference report, and determining whether the cosine similarity is greater than a preset similarity threshold; if the cosine similarity is greater than the similarity threshold, then re-executing the random mask sampling operation and the preset sentence-level splitting operation to generate a corresponding target non-preference report; if the cosine similarity is less than or equal to the similarity threshold, then determining the corresponding target non-preference report based on the initial non-preference sample, and performing a preset direct preference optimization operation on the basic fine-tuning model based on the target non-preference report, the composite loss function, and the preference pair dataset to generate the target CT report.

[0012] Optionally, in one embodiment of this application, the mathematical expression of the composite loss function is:

[0013] in, This represents the direct preference optimization loss function; This represents the negative log-likelihood regularization loss function; The weighting coefficients represent the weights that balance the direct preference optimization loss function and the negative log-likelihood regularization loss.

[0014] A second aspect of this application provides a CT report direct preference optimization device based on visual attention masking, comprising: a mask sampling module for acquiring various three-dimensional CT image data, fine-tuning a pre-built basic model using preprocessed data of the various three-dimensional CT image data to generate a corresponding basic fine-tuning model, and encoding two-dimensional CT image slices in the preprocessed data into multiple embedding vectors, and performing random mask sampling on the multiple embedding vectors to obtain a masked CT image; and a reference guidance module for performing preset sentence splitting and truncation operations on a preset actual preference report to generate a truncation prefix for each independent sentence, and optimizing the masked CT image. The CT image and the truncated prefix of the current independent sentence are input into the basic fine-tuning model to generate a new sentence suffix. Based on the new sentence suffix and the truncated prefix, a preset sentence splicing and combination operation is performed to generate an initial non-preference sample that meets preset consistency requirements. The preference optimization module is used to perform a preset semantic similarity filtering operation on the initial non-preference sample to obtain the corresponding target non-preference report. Based on the two-dimensional CT image slice, the actual preference report, and the target non-preference report, and combined with a preset composite loss function, a preset direct preference optimization operation is performed on the basic fine-tuning model to generate a target CT report that meets preset quality requirements.

[0015] Optionally, in one embodiment of this application, the mask sampling module includes: an interpolation unit, used to unify the resolution and pixel spacing parameters of the various three-dimensional CT image data to obtain corresponding parameterized CT image data, and to extract multiple two-dimensional CT image slices from the parameterized CT image data at a target interval using a preset cubic spline interpolation strategy; a normalization unit, used to unify the format and character encoding of the real report text corresponding to the multiple two-dimensional CT image slices to obtain corresponding standard report text, and to determine the preprocessed data based on the multiple two-dimensional CT image slices and the standard report text, so as to perform a preset supervised fine-tuning operation on the basic model through the preprocessed data to obtain the basic fine-tuning model; and an encoding unit, used to encode the two-dimensional CT image slices in the preprocessed data into the multiple embedding vectors through the visual encoder of the basic fine-tuning model, and to perform random mask sampling operation on the multiple embedding vectors based on a preset mask threshold to obtain the masked CT image.

[0016] Optionally, in one embodiment of this application, the reference guidance module includes: a splitting unit, configured to perform a preset sentence-level splitting operation on the actual preference report to generate multiple independent sentences, and determine the truncation position of each independent sentence, so as to split the corresponding independent sentence into a truncation prefix and an actual suffix according to the truncation position; a generation unit, configured to input the masked CT image, the truncation prefix of the current independent sentence, and the generated answers of all independent sentences before the current independent sentence into the basic fine-tuning model to generate a new independent sentence suffix; and a splicing unit, configured to splice the truncation prefix of the current independent sentence with the new independent sentence suffix to obtain a spliced ​​independent sentence of the current independent sentence, and combine the spliced ​​independent sentences of each independent sentence to construct an initial non-preference sample that meets the preset consistency requirement with the actual preference report.

[0017] Optionally, in one embodiment of this application, the preference optimization module includes: a calculation unit, configured to calculate the cosine similarity between the initial non-preference sample and the actual preference report, and determine whether the cosine similarity is greater than a preset similarity threshold; a first analysis unit, configured to re-execute the random mask sampling operation and the preset sentence-level splitting operation if the cosine similarity is greater than the similarity threshold, to generate the corresponding target non-preference report; and a second analysis unit, configured to determine the corresponding target non-preference report based on the initial non-preference sample if the cosine similarity is less than or equal to the similarity threshold, and perform a preset direct preference optimization operation on the basic fine-tuning model based on the target non-preference report, the composite loss function, and the preference pair dataset, to generate the target CT report.

[0018] Optionally, in one embodiment of this application, the mathematical expression of the composite loss function is:

[0019] in, This represents the direct preference optimization loss function; This represents the negative log-likelihood regularization loss function; The weighting coefficients represent the weights that balance the direct preference optimization loss function and the negative log-likelihood regularization loss.

[0020] A third aspect of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the CT report direct preference optimization method based on visual attention mask as described in the above embodiments.

[0021] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for direct preference optimization of CT reports based on visual attention masks.

[0022] A fifth aspect of this application provides a computer program product, including a computer program that is executed to implement the above-described method for direct preference optimization of CT reports based on visual attention masks.

[0023] Therefore, the embodiments of this application have the following beneficial effects: The embodiments of this application can acquire various 3D CT image data, fine-tune a pre-built basic model using preprocessed data of various 3D CT image data, generate a corresponding basic fine-tuning model, and encode 2D CT image slices in the preprocessed data into multiple embedding vectors, and perform random masking sampling on multiple embedding vectors to obtain a masked CT image; perform preset sentence splitting and truncation operations on the preset actual preference report to generate a truncated prefix for each independent sentence, and input the masked CT image and the truncated prefix of the current independent sentence into the basic fine-tuning model to generate a new sentence suffix, and perform a preset sentence splicing and combination operation based on the new sentence suffix and truncated prefix to generate an initial non-preference sample that meets preset consistency requirements; perform a preset semantic similarity filtering operation on the initial non-preference sample to obtain the corresponding target non-preference report, and perform a preset direct preference optimization operation on the basic fine-tuning model based on the 2D CT image slices, the actual preference report, and the target non-preference report, combined with a preset composite loss function, to generate a target CT report that meets preset quality requirements. This application actively blocks access to visual information through a visual attention masking mechanism. This allows the model to generate hallucinatory outputs based on its prior linguistic knowledge, thus forming non-preferred reports. This effectively suppresses false positives and false negatives. Furthermore, a reference-guided generation strategy ensures a high degree of syntactic consistency between preferred and non-preferred reports, allowing DPO training to focus on distinguishing factual content from hallucinatory descriptions, avoiding shortcut learning, and significantly improving hallucination suppression efficiency. Therefore, this addresses the problems of existing technologies that over-rely on prior linguistic knowledge leading to factual hallucinations, lack of mechanisms to construct non-preferred samples incorporating visual evidence, and inability to improve clinical accuracy without requiring extensive additional labeled data.

[0024] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0025] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 A flowchart of a CT report direct preference optimization method based on visual attention masking is provided for one embodiment of this application; Figure 2 This is a schematic diagram of the logical architecture of a CT report direct preference optimization method based on visual attention mask according to an embodiment of this application; Figure 3 A schematic diagram illustrating the execution logic of a CT report direct preference optimization method based on visual attention masking, provided as an embodiment of this application; Figure 4 This is an example diagram of a CT report direct preference optimization device based on visual attention mask according to an embodiment of this application; Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0026] Among them, 10-CT report direct preference optimization device based on visual attention mask; 100-mask sampling module, 200-reference guidance module, 300-preference optimization module; 501-memory, 502-processor, 503-communication interface. Detailed Implementation

[0027] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0028] The following describes, with reference to the accompanying drawings, a method and apparatus for direct preference optimization of CT reports based on visual attention masks, according to embodiments of this application. To address the problems mentioned in the background, this application provides a direct preference optimization method for CT reports based on visual attention masks. In this method, multiple types of 3D CT image data are acquired, and a pre-constructed basic model is fine-tuned using preprocessed data from these 3D CT images to generate a corresponding basic fine-tuning model. Two-dimensional CT image slices in the preprocessed data are encoded into multiple embedding vectors, and random masking sampling is performed on these embedding vectors to obtain a masked CT image. A preset sentence splitting and truncation operation is performed on the preset actual preference report to generate a truncated prefix for each independent sentence. The masked CT image and the truncated prefix of the current independent sentence are input into the basic fine-tuning model to generate a new sentence suffix. Based on the new sentence suffix and the truncated prefix, a preset sentence concatenation and combination operation is performed to generate an initial non-preference sample that meets preset consistency requirements. A preset semantic similarity filtering operation is performed on the initial non-preference sample to obtain the corresponding target non-preference report. Based on the two-dimensional CT image slices, the actual preference report, and the target non-preference report, and combined with a preset composite loss function, a preset direct preference optimization operation is performed on the basic fine-tuning model to generate a target CT report that meets preset quality requirements. This application actively blocks access to visual information through a visual attention masking mechanism. This allows the model to generate hallucinatory outputs based on its prior linguistic knowledge, thus forming non-preferred reports. This effectively suppresses false positives and false negatives. Furthermore, a reference-guided generation strategy ensures a high degree of syntactic consistency between preferred and non-preferred reports, allowing DPO training to focus on distinguishing factual content from hallucinatory descriptions, avoiding shortcut learning, and significantly improving hallucination suppression efficiency. Therefore, this addresses the problems of existing technologies that over-rely on prior linguistic knowledge leading to factual hallucinations, lack of mechanisms to construct non-preferred samples incorporating visual evidence, and inability to improve clinical accuracy without requiring extensive additional labeled data.

[0029] Specifically, Figure 1 This is a flowchart of a direct preference optimization method for CT reports based on visual attention masks, provided as an embodiment of this application.

[0030] like Figure 1 As shown, the CT report direct preference optimization method based on visual attention masking includes the following steps: In step S101, various three-dimensional CT image data are acquired, and the pre-constructed basic model is fine-tuned using the pre-processed data of various three-dimensional CT image data to generate the corresponding basic fine-tuning model. The two-dimensional CT image slices in the pre-processed data are encoded into multiple embedding vectors, and random mask sampling is performed on the multiple embedding vectors to obtain the masked CT image.

[0031] In this embodiment, a multimodal large language model can be used as the base model. The base model is then fine-tuned under supervision using mixed CT report data (i.e., CT-report data pairs) and public radiology datasets such as PubMedVision, ROCOv2, and PMC-VQA. This enables the base model to generate coherent reports in diverse clinical scenarios. The base model after supervision and fine-tuning (i.e., the base fine-tuned model) is then used as the strategy model for subsequent hallucination induction and the DPO framework.

[0032] Secondly, in this application embodiment, the image token embedding vector (i.e., image token embedding vector) of the input CT image (i.e., two-dimensional CT image slice encoding) can be randomly masked to block the acquisition of visual information by the fine-tuned model, thereby forcing the model to generate a non-biased report containing factual illusions by relying solely on prior linguistic knowledge.

[0033] Optionally, in one embodiment of this application, multiple types of 3D CT image data are acquired, and a pre-constructed basic model is fine-tuned using preprocessed data of the multiple 3D CT image data to generate a corresponding basic fine-tuning model. Two-dimensional CT image slices in the preprocessed data are encoded into multiple embedding vectors, and random mask sampling is performed on the multiple embedding vectors to obtain a masked CT image. This includes: unifying the resolution and pixel spacing parameters of the multiple 3D CT image data to obtain corresponding parameterized CT image data; using a preset cubic spline interpolation strategy to extract multiple two-dimensional CT image slices from the parameterized CT image data at target intervals; standardizing the format and character encoding of the real report text corresponding to the multiple two-dimensional CT image slices to obtain corresponding standard report text; determining preprocessed data based on the multiple two-dimensional CT image slices and the standard report text; performing preset supervised fine-tuning operations on the basic model using the preprocessed data to obtain a basic fine-tuning model; and encoding the two-dimensional CT image slices in the preprocessed data into multiple embedding vectors using the visual encoder of the basic fine-tuning model, and performing random mask sampling on the multiple embedding vectors based on a preset mask threshold to obtain a masked CT image.

[0034] Specifically, embodiments of this application first collect three-dimensional CT image data and corresponding real report texts, i.e., CT-report data pairs. The CT image data comes from clinically acquired chest CT scans, covering multi-center data with different equipment and scanning parameters; the real report text is written by radiologists and includes detailed descriptions of anatomical structures such as the lungs, heart, and mediastinum, as well as characteristic information such as the location, size, shape, and density of lesions.

[0035] Secondly, the embodiments of this application can unify the resolution and pixel spacing parameters of three-dimensional CT image data to obtain corresponding parameterized CT image data. A cubic spline interpolation method is then used to extract two-dimensional CT image slices from the parameterized CT image data at 3mm intervals (i.e., the target interval), ensuring the comparability of images from different sources. The format and character encoding of the actual report text corresponding to multiple two-dimensional CT image slices are standardized to remove irrelevant characters and error messages.

[0036] It should be noted that the embodiments of this application can use the pre-trained general vision-language model Qwen3-VL-8B as the base model mentioned above. This model natively supports region-aware understanding, providing the necessary foundation for predicting bounding boxes. The structure of Qwen3-VL-8B includes three core parts: a visual encoder, a language model, and a connector. The visual encoder is typically a Vision Transformer. The connector is a perceptron-resampler module used to map the image feature vectors output by the visual encoder to a semantic space that the language model can understand. The language model is a large language model based on the Transformer architecture, responsible for understanding and generating CT report text.

[0037] It should be noted that although the Qwen3-VL-8B is preferred as the base model in the embodiments of this application, in actual implementation, those skilled in the art can also use other general models with strong visual-language understanding capabilities and support for region awareness, such as InternVL, Fuyu series, or an architecture composed of a dedicated visual encoder and a large language model spliced ​​together by a trainable adapter, all of which can be used as alternatives, and no specific limitation is made here.

[0038] Therefore, in this embodiment, preprocessed CT image slices (i.e., two-dimensional CT image slices) can be input into the aforementioned visual encoder to extract image features and convert them into an image token embedding sequence. Each embedding vector (i.e., image token embedding vector) in this sequence corresponds to a local region in the image, forming a visual representation that can be processed by a multimodal large language model. Specifically, as one possible implementation, the input CT image (i.e., two-dimensional CT image slice) in this embodiment is encoded by the aforementioned visual encoder as follows: 1 embedding vector (i.e., image token embedding vector) Subsequently, embodiments of this application can be based on a uniform distribution U(0,1) and a preset mask threshold. Random sampling generates a Boolean mask. The calculation method for each element in the mask is as follows:

[0039] in, , Indicates the first The image embedding is masked; a preset mask threshold is used. The value can be set to 0.99.

[0040] Understandably, existing direct preference optimization methods primarily operate at the text level or rely on external knowledge retrieval when constructing suboptimal samples, failing to address the root cause of the model's neglect of visual evidence. This application's embodiments actively block access to visual information through a visual attention masking mechanism, inducing the model to rely on linguistic priors to generate suboptimal samples containing factual illusions. This integrates visual evidence guidance into the preference pair construction process, enabling the model to more accurately learn the correspondence between visual and textual information, thereby effectively suppressing false positive and false negative illusions.

[0041] In step S102, the preset actual preference report is subjected to preset sentence splitting and truncation operations to generate a truncated prefix for each independent sentence. The masked CT image and the truncated prefix of the current independent sentence are input into the basic fine-tuning model to generate a new sentence suffix. Based on the new sentence suffix and the truncated prefix, a preset sentence splicing and combination operation is performed to generate an initial non-preference sample that meets the preset consistency requirements.

[0042] Furthermore, embodiments of this application can split the real preference CT report to obtain multiple independent sentences, and truncate each independent sentence. The masked CT image, the truncated prefix of the current independent sentence, and all previously generated independent sentences are used as model input to guide the multimodal large language model to generate new sentence suffixes. Subsequently, embodiments of this application can concatenate and combine the truncated prefix and the newly generated sentence suffixes to construct an initial non-preference sample that meets the consistency requirements.

[0043] Optionally, in one embodiment of this application, a preset sentence splitting and truncation operation is performed on the preset actual preference report to generate a truncated prefix for each independent sentence. The masked CT image and the truncated prefix of the current independent sentence are input into the basic fine-tuning model to generate a new sentence suffix. Based on the new sentence suffix and the truncated prefix, a preset sentence concatenation and combination operation is performed to generate an initial non-preference sample that meets the preset consistency requirements. This includes: performing a preset sentence-level splitting operation on the actual preference report to generate multiple independent sentences, and determining the truncation position of each independent sentence so as to split the corresponding independent sentence into a truncated prefix and an actual suffix according to the truncation position; inputting the masked CT image, the truncated prefix of the current independent sentence, and the generated answers of all independent sentences before the current independent sentence into the basic fine-tuning model to generate a new independent sentence suffix; concatenating the truncated prefix of the current independent sentence with the new independent sentence suffix to obtain a concatenated independent sentence of the current independent sentence, and combining the concatenated independent sentences of each independent sentence to construct an initial non-preference sample that meets the preset consistency requirements with the actual preference report.

[0044] As one possible approach, embodiments of this application can split the real preference CT report (i.e., the actual preference report) at the sentence level to obtain multiple independent sentences. Within each independent sentence, a truncation position is randomly selected, and the sentence is split into a truncated prefix and an actual suffix based on this truncation position. Secondly, embodiments of this application can use the masked CT image, the truncated prefix of the current independent sentence, and all previously generated independent sentences as model input to guide the multimodal large language model to continue writing new sentence suffixes. Finally, embodiments of this application concatenate the truncated prefix with the newly generated sentence suffix to obtain a concatenated independent sentence of the current independent sentence, and combine all processed independent sentences (i.e., concatenated independent sentences) to construct an initial non-preference sample that maintains a high degree of consistency with the preference report in overall syntactic structure (i.e., meets the preset consistency requirements).

[0045] Specifically, the true preference CT report (i.e., the actual preference report) is broken down into Each independent sentence For the segmented independent sentences, the multimodal large language model is based on the masked CT image input. truncated prefix and all sentences generated before the current independent sentence. Perform autoregressive inference to generate new sentence suffixes (i.e., new illusion suffixes). The generation process of new sentence suffixes follows the distribution below:

[0046] in, This represents the generation distribution of a multimodal large language model.

[0047] It should be noted that the process of generating the new sentence suffixes will continue until the model outputs a period or the end-of-sequence word. <eos>Until then; subsequently, embodiments of this application may truncate the prefix. With the generated new sentence suffix The sentences are then pieced together to form complete unbiased sentences, which in turn form the corresponding initial unbiased samples.

[0048] Because existing technologies often exhibit syntactic structural differences between preferred and non-preferred reports, models tend to judge preferences based on sentence structure rather than content authenticity, thus weakening the hallucination suppression effect. Therefore, this application's embodiments utilize a guided generation strategy to ensure a high degree of consistency in syntactic structure between preferred and non-preferred reports. This allows DPO training to focus on distinguishing between factual content and hallucinatory descriptions, avoiding shortcut learning and significantly improving hallucination suppression efficiency.

[0049] In step S103, a preset semantic similarity filtering operation is performed on the initial non-preference samples to obtain the corresponding target non-preference report. Based on the two-dimensional CT image slices, the actual preference report and the target non-preference report, and combined with the preset composite loss function, a preset direct preference optimization operation is performed on the basic fine-tuning model to generate a target CT report that meets the preset quality requirements.

[0050] After that, as Figure 2 As shown, embodiments of this application can introduce a filtering mechanism based on sentence-level semantic similarity to calculate the cosine similarity between the generated initial non-preference sample and the original preference report (i.e., the actual preference report). If the cosine similarity exceeds a preset threshold, masking and other operations are performed again to filter out samples with too small a difference, and finally obtain a non-preference report with substantial hallucination content.

[0051] Secondly, embodiments of this application can utilize masked CT images, actual preference reports, and non-preference reports to construct corresponding preference pair datasets. By combining a composite loss function consisting of a direct preference optimization loss function and a negative log-likelihood regularization loss function, direct preference optimization training is performed on the supervised fine-tuned multimodal large language model (i.e., the basic fine-tuned model) to generate high-quality, low-illusion CT reports.

[0052] Optionally, in one embodiment of this application, a preset semantic similarity filtering operation is performed on the initial unpreference sample to obtain the corresponding target unpreference report. Based on the two-dimensional CT image slices, the actual preference report, and the target unpreference report, and combined with a preset composite loss function, a preset direct preference optimization operation is performed on the basic fine-tuning model to generate a target CT report that meets preset quality requirements. This includes: calculating the cosine similarity between the initial unpreference sample and the actual preference report, and determining whether the cosine similarity is greater than a preset similarity threshold; if the cosine similarity is greater than the similarity threshold, then the random mask sampling operation and the preset sentence-level splitting operation are re-executed to generate the corresponding target unpreference report; if the cosine similarity is less than or equal to the similarity threshold, then the corresponding target unpreference report is determined according to the initial unpreference sample, and based on the target unpreference report, the composite loss function, and the preference pair dataset, a preset direct preference optimization operation is performed on the basic fine-tuning model to generate the target CT report.

[0053] It should be noted that the embodiments of this application may use a pre-trained sentence embedding model (such as all-mpnet-base-v2) to compute the initial unbiased samples. Compared with actual preference reports The cosine similarity between them, and the preset similarity threshold. The cosine similarity can be set to 0.97. If the cosine similarity exceeds this threshold, the attention mask m and sentence splitting position are resampled. The maximum number of attempts (i.e., the upper limit of attempts) N generated by this resampling can be set to 5. If the cosine similarity requirement is still not met after 5 attempts (i.e., the cosine similarity is greater than the similarity threshold), the final non-preference report will no longer refer to the original preference report.

[0054] Optionally, in one embodiment of this application, the mathematical expression of the composite loss function is:

[0055] in, This represents the direct preference optimization loss function; This represents the negative log-likelihood regularization loss function; This represents the weighting coefficients for balancing the direct preference optimization loss function and the negative log-likelihood regularization loss.

[0056] Secondly, during the direct preference optimization training process, the total loss function is obtained by weighted summation of the direct preference optimization loss function and the negative log-likelihood regularization loss function, as shown in the following equation:

[0057] in, This represents the direct preference optimization loss function; This represents the negative log-likelihood regularization loss function; This represents the weighting coefficients for balancing the direct preference optimization loss function and the negative log-likelihood regularization loss.

[0058] The mathematical expression for the direct preference optimization loss function is:

[0059] The mathematical expression for the negative log-likelihood regularization loss function is:

[0060] Where D represents the preference pair for the dataset; Indicates a reference strategy; This represents the temperature hyperparameter in the direct preference optimization loss function, which can be set to 0.2. The weighting coefficient used to balance the direct preference optimization loss function and the negative log-likelihood regularization loss function can be selected as 0.1; This represents triplet data sampled from the preference pair dataset D. The mathematical expectation.

[0061] Understandably, traditional methods rely on large-scale labeled data to construct preference datasets, which is difficult to promote and apply given the scarcity of 3D CT image labeled data. The embodiments of this application do not require additional manual labeling. They can automatically construct high-quality preference data pairs using only visual attention masks and reference guidance, and ensure sample quality through a semantic similarity filtering mechanism. This significantly reduces the dependence on labeled data and improves the generalizability of the method.

[0062] The execution logic of the CT report direct preference optimization method based on visual attention mask of this application will be described below with reference to the accompanying drawings.

[0063] Figure 3 This is a schematic diagram illustrating the execution logic of the CT report direct preference optimization method based on visual attention masking proposed in this application. Figure 3 As shown, the execution process of the CT report direct preference optimization method based on visual attention mask in this application is as follows: S301: Supervised fine-tuning of the base model, i.e., the multimodal large language model; S302: Randomly mask the image word embedding vectors of CT images (i.e., 2D CT image slices) so that the model backs down when visual signals are insufficient and relies only on its parameterized linguistic prior knowledge, thereby generating a non-biased report containing factual illusions. S303: Based on the reference-guided generation strategy, the report generates an initial non-preference sample that is highly consistent with the preference report in terms of overall syntactic structure; S304: Calculate the cosine similarity between the initial non-preference sample and the original preference report, and perform filtering based on the cosine similarity and similarity threshold to obtain the final non-preference report (i.e. the target non-preference report). If the cosine similarity is greater than the similarity threshold and the upper limit of attempts has not been reached, proceed to S302; otherwise, proceed to S305. S305: Using a preference pair dataset containing CT images, original preference reports, and final non-preference reports, the direct preference optimization loss function and the negative log-likelihood regularization loss function are combined to perform direct preference optimization on a supervised fine-tuned multimodal large language model, generating high-quality, low-illusion CT reports.

[0064] According to the CT report direct preference optimization method based on visual attention masking proposed in this application, a pre-constructed basic model is fine-tuned using pre-processed data of various 3D CT images to generate a corresponding basic fine-tuning model. Two-dimensional CT image slices in the pre-processed data are encoded into multiple embedding vectors, and random masking sampling is performed on these embedding vectors to obtain a masked CT image. A preset sentence splitting and truncation operation is performed on the preset actual preference report to generate a truncated prefix for each independent sentence. The masked CT image and the truncated prefix of the current independent sentence are input into the basic fine-tuning model to generate a new sentence suffix. Based on the new sentence suffix and the truncated prefix, a preset sentence splicing and combination operation is performed to generate an initial non-preference sample that meets preset consistency requirements. A preset semantic similarity filtering operation is performed on the initial non-preference sample to obtain the corresponding target non-preference report. Based on the two-dimensional CT image slices, the actual preference report, and the target non-preference report, and combined with a preset composite loss function, a preset direct preference optimization operation is performed on the basic fine-tuning model to generate a target CT report that meets preset quality requirements. This application actively blocks access to visual information through a visual attention masking mechanism, enabling the model to generate hallucinatory outputs based on its own linguistic prior knowledge when vision is blocked. This results in non-optimal reports, effectively suppressing false positives and false negatives. Furthermore, through a reference-guided generation strategy, it ensures that the optimal and non-optimal reports maintain a high degree of consistency in syntactic structure. This allows DPO training to focus on distinguishing between factual content and hallucinatory descriptions, avoiding shortcut learning and significantly improving hallucination suppression efficiency.

[0065] Secondly, with reference to the accompanying drawings, a CT report direct preference optimization device based on visual attention masking according to an embodiment of this application is described.

[0066] Figure 4 This is a block diagram of a CT report direct preference optimization device based on visual attention mask according to an embodiment of this application.

[0067] like Figure 4 As shown, the CT report direct preference optimization device 10 based on visual attention mask includes: a mask sampling module 100, a reference guidance module 200, and a preference optimization module 300.

[0068] The mask sampling module 100 is used to acquire various three-dimensional CT image data, and fine-tune the pre-built basic model through the pre-processed data of various three-dimensional CT image data to generate the corresponding basic fine-tuning model. It also encodes the two-dimensional CT image slices in the pre-processed data into multiple embedding vectors and performs random mask sampling operation on the multiple embedding vectors to obtain the masked CT image.

[0069] The reference guidance module 200 is used to perform preset sentence splitting and truncation operations on the preset actual preference report to generate a truncated prefix for each independent sentence. The masked CT image and the truncated prefix of the current independent sentence are input into the basic fine-tuning model to generate a new sentence suffix. Based on the new sentence suffix and the truncated prefix, a preset sentence splicing and combination operation is performed to generate an initial non-preference sample that meets the preset consistency requirements.

[0070] The preference optimization module 300 is used to perform a preset semantic similarity filtering operation on the initial non-preference samples to obtain the corresponding target non-preference report. Based on the two-dimensional CT image slices, the actual preference report and the target non-preference report, and combined with the preset composite loss function, the module performs a preset direct preference optimization operation on the basic fine-tuning model to generate a target CT report that meets the preset quality requirements.

[0071] Optionally, in one embodiment of this application, the mask sampling module 100 includes: an interpolation unit, a normalization unit, and an encoding unit.

[0072] The interpolation unit is used to unify the resolution and pixel spacing parameters of various three-dimensional CT image data to obtain corresponding parameter-unified CT image data. It also uses a preset cubic spline interpolation strategy to extract multiple two-dimensional CT image slices from the parameter-unified CT image data at target intervals.

[0073] The normalization unit is used to standardize the format and character encoding of the real report text corresponding to multiple 2D CT image slices to obtain the corresponding standard report text. Based on multiple 2D CT image slices and standard report texts, preprocessing data is determined so that the basic model can be subjected to preset supervised fine-tuning operations through the preprocessing data to obtain the basic fine-tuning model.

[0074] The encoding unit is used to encode two-dimensional CT image slices in the preprocessed data into multiple embedding vectors through a visual encoder of a basic fine-tuning model, and to perform random mask sampling on the multiple embedding vectors based on a preset mask threshold to obtain a masked CT image.

[0075] Optionally, in one embodiment of this application, the reference guidance module 200 includes: a splitting unit, a generating unit, and a splicing unit.

[0076] The splitting unit is used to perform a preset sentence-level splitting operation on the actual preference report to generate multiple independent sentences and determine the truncation position of each independent sentence so as to split the corresponding independent sentence into a truncation prefix and an actual suffix according to the truncation position.

[0077] The generation unit is used to input the masked CT image, the truncated prefix of the current independent sentence, and the generated answers of all independent sentences preceding the current independent sentence into the basic fine-tuning model to generate new independent sentence suffixes.

[0078] The splicing unit is used to splice the truncated prefix of the current independent sentence with the suffix of the new independent sentence to obtain the spliced ​​independent sentence of the current independent sentence, and to combine the spliced ​​independent sentences of each independent sentence to construct an initial non-preference sample that meets the preset consistency requirements with the actual preference report.

[0079] Optionally, in one embodiment of this application, the preference optimization module 300 includes: a calculation unit, a first analysis unit, and a second analysis unit.

[0080] The calculation unit is used to calculate the cosine similarity between the initial non-preference sample and the actual preference report, and to determine whether the cosine similarity is greater than a preset similarity threshold.

[0081] The first analysis unit is used to re-execute the random mask sampling operation and the preset sentence-level splitting operation if the cosine similarity is greater than the similarity threshold, so as to generate the corresponding target non-preference report.

[0082] The second analysis unit is used to determine the corresponding target non-preference report based on the initial non-preference sample if the cosine similarity is less than or equal to the similarity threshold, and to perform a preset direct preference optimization operation on the basic fine-tuning model based on the target non-preference report, the composite loss function and the preference pair dataset to generate the target CT report.

[0083] Optionally, in one embodiment of this application, the mathematical expression of the composite loss function is:

[0084] in, This represents the direct preference optimization loss function; This represents the negative log-likelihood regularization loss function; This represents the weighting coefficients for balancing the direct preference optimization loss function and the negative log-likelihood regularization loss.

[0085] It should be noted that the foregoing explanation of the embodiment of the CT report direct preference optimization method based on visual attention mask also applies to the CT report direct preference optimization device based on visual attention mask in this embodiment, and will not be repeated here.

[0086] The CT report direct preference optimization device based on visual attention masking proposed in this application includes a mask sampling module 100, which is used to acquire various three-dimensional CT image data, fine-tune a pre-built basic model through preprocessed data of various three-dimensional CT image data, generate a corresponding basic fine-tuning model, and encode two-dimensional CT image slices in the preprocessed data into multiple embedding vectors, and perform random mask sampling operation on the multiple embedding vectors to obtain a masked CT image; and a reference guidance module 200, which is used to perform preset sentence splitting and truncation operations on the preset actual preference report to generate a truncation prefix for each independent sentence, and mask the actual preference report. The CT image after encoding and the truncated prefix of the current independent sentence are input into the basic fine-tuning model to generate a new sentence suffix. Based on the new sentence suffix and the truncated prefix, a preset sentence concatenation and combination operation is performed to generate an initial non-preferred sample that meets the preset consistency requirements. The preference optimization module 300 is used to perform a preset semantic similarity filtering operation on the initial non-preferred sample to obtain the corresponding target non-preferred report. Based on the two-dimensional CT image slice, the actual preference report, and the target non-preferred report, and combined with a preset composite loss function, a preset direct preference optimization operation is performed on the basic fine-tuning model to generate a target CT report that meets the preset quality requirements. This application actively blocks access to visual information through a visual attention masking mechanism, so that the model generates hallucinatory outputs based on its own linguistic prior knowledge when vision is blocked, thereby forming a non-optimal report. This effectively suppresses false positive and false negative hallucinations. Through a reference-guided generation strategy, it ensures that the optimal and non-optimal reports maintain a high degree of consistency in syntactic structure, so that DPO training focuses on distinguishing factual content from hallucinatory descriptions, avoiding shortcut learning, and significantly improving the efficiency of hallucination suppression.

[0087] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may include: The memory 501, the processor 502, and the computer program stored on the memory 501 and capable of running on the processor 502.

[0088] When the processor 502 executes the program, it implements the CT report direct preference optimization method based on visual attention mask provided in the above embodiments.

[0089] Furthermore, electronic devices also include: Communication interface 503 is used for communication between memory 501 and processor 502.

[0090] The memory 501 is used to store computer programs that can run on the processor 502.

[0091] The memory 501 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage.

[0092] If the memory 501, processor 502, and communication interface 503 are implemented independently, then the communication interface 503, memory 501, and processor 502 can be interconnected via a bus to complete communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a PCI bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 5 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0093] Optionally, in a specific implementation, if the memory 501, processor 502, and communication interface 503 are integrated on a single chip, then the memory 501, processor 502, and communication interface 503 can communicate with each other through an internal interface.

[0094] Processor 502 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of this application.

[0095] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described CT report direct preference optimization method based on visual attention mask.

[0096] This application also provides a computer program product, including a computer program, which, when executed, is used to implement the above-described CT report direct preference optimization method based on visual attention mask.

[0097] In the description of this specification, the references to "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0098] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0099] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0100] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0101] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. If implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0102] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0103] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0104] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.< / eos>

Claims

1. A method for optimizing direct preferences in CT reports based on visual attention masks, characterized in that, Includes the following steps: Multiple types of 3D CT image data are acquired, and a pre-constructed basic model is fine-tuned using the pre-processed data of the multiple types of 3D CT image data to generate a corresponding basic fine-tuned model. The 2D CT image slices in the pre-processed data are encoded into multiple embedding vectors, and random mask sampling is performed on the multiple embedding vectors to obtain the masked CT image. The preset actual preference report is subjected to preset sentence splitting and truncation operations to generate a truncation prefix for each independent sentence. The masked CT image and the truncation prefix of the current independent sentence are input into the basic fine-tuning model to generate a new sentence suffix. Based on the new sentence suffix and the truncation prefix, a preset sentence splicing and combination operation is performed to generate an initial non-preference sample that meets the preset consistency requirements. The initial non-preference samples are subjected to a preset semantic similarity filtering operation to obtain the corresponding target non-preference report. Based on the two-dimensional CT image slices, the actual preference report, and the target non-preference report, and combined with a preset composite loss function, the basic fine-tuning model is subjected to a preset direct preference optimization operation to generate a target CT report that meets preset quality requirements.

2. The CT report direct preference optimization method based on visual attention mask according to claim 1, characterized in that, The process involves acquiring multiple types of 3D CT image data, fine-tuning a pre-built basic model using preprocessed data from these 3D CT images to generate a corresponding basic fine-tuned model, and encoding 2D CT image slices in the preprocessed data into multiple embedding vectors. Random masking sampling is then performed on these embedding vectors to obtain a masked CT image. The resolution and pixel spacing parameters of the various three-dimensional CT image data are unified to obtain corresponding parameter unified CT image data. A preset cubic spline interpolation strategy is used to extract multiple two-dimensional CT image slices from the parameter unified CT image data at target intervals. The format and character encoding of the real report texts corresponding to the multiple two-dimensional CT image slices are standardized to obtain the corresponding standard report texts. Based on the multiple two-dimensional CT image slices and the standard report texts, the preprocessed data is determined, and the basic model is subjected to a preset supervised fine-tuning operation through the preprocessed data to obtain the basic fine-tuning model. The visual encoder of the basic fine-tuning model encodes the two-dimensional CT image slices in the preprocessed data into the multiple embedding vectors, and performs random mask sampling on the multiple embedding vectors based on a preset mask threshold to obtain the masked CT image.

3. The CT report direct preference optimization method based on visual attention mask according to claim 1, characterized in that, The process involves performing preset sentence splitting and truncation operations on the preset actual preference report to generate a truncated prefix for each independent sentence. The masked CT image and the truncated prefix of the current independent sentence are then input into the basic fine-tuning model to generate a new sentence suffix. Based on the new sentence suffix and the truncated prefix, a preset sentence concatenation and combination operation is performed to generate an initial non-preference sample that meets preset consistency requirements. This includes: The actual preference report is split into multiple independent sentences by a preset sentence-level splitting operation, and the truncation position of each independent sentence is determined so that the corresponding independent sentence is split into a truncation prefix and an actual suffix according to the truncation position. The masked CT image, the truncated prefix of the current independent sentence, and the generated responses of all independent sentences preceding the current independent sentence are input into the basic fine-tuning model to generate new independent sentence suffixes. The truncated prefix of the current independent sentence is concatenated with the suffix of the new independent sentence to obtain the concatenated independent sentence of the current independent sentence. The concatenated independent sentences of each independent sentence are combined to construct an initial non-preference sample that meets the preset consistency requirement with the actual preference report.

4. The CT report direct preference optimization method based on visual attention mask according to claim 1, characterized in that, The process of performing a preset semantic similarity filtering operation on the initial non-preferred samples to obtain the corresponding target non-preferred report, and then performing a preset direct preference optimization operation on the basic fine-tuning model based on the two-dimensional CT image slices, the actual preference report, and the target non-preferred report, combined with a preset composite loss function, to generate a target CT report that meets preset quality requirements, includes: Calculate the cosine similarity between the initial non-preference sample and the actual preference report, and determine whether the cosine similarity is greater than a preset similarity threshold; If the cosine similarity is greater than the similarity threshold, the random mask sampling operation and the preset sentence-level splitting operation are re-executed to generate the corresponding target non-preference report; If the cosine similarity is less than or equal to the similarity threshold, the corresponding target non-preference report is determined based on the initial non-preference sample, and a preset direct preference optimization operation is performed on the basic fine-tuning model based on the target non-preference report, the composite loss function, and the preference pair dataset to generate the target CT report.

5. The CT report direct preference optimization method based on visual attention mask according to claim 4, characterized in that, The mathematical expression for the composite loss function is: in, This represents the direct preference optimization loss function; This represents the negative log-likelihood regularization loss function; The weighting coefficients represent the weights that balance the direct preference optimization loss function and the negative log-likelihood regularization loss.

6. A CT report direct preference optimization device based on visual attention masking, characterized in that, include: The mask sampling module is used to acquire various three-dimensional CT image data, and fine-tune the pre-built basic model through the pre-processed data of the various three-dimensional CT image data to generate the corresponding basic fine-tuning model. It also encodes the two-dimensional CT image slices in the pre-processed data into multiple embedding vectors, and performs random mask sampling operation on the multiple embedding vectors to obtain the masked CT image. The reference guidance module is used to perform preset sentence splitting and truncation operations on the preset actual preference report to generate a truncation prefix for each independent sentence. The masked CT image and the truncation prefix of the current independent sentence are input into the basic fine-tuning model to generate a new sentence suffix. Based on the new sentence suffix and the truncation prefix, a preset sentence splicing and combination operation is performed to generate an initial non-preference sample that meets the preset consistency requirements. The preference optimization module is used to perform a preset semantic similarity filtering operation on the initial non-preference samples to obtain the corresponding target non-preference report. Based on the two-dimensional CT image slices, the actual preference report and the target non-preference report, and combined with a preset composite loss function, the module performs a preset direct preference optimization operation on the basic fine-tuning model to generate a target CT report that meets preset quality requirements.

7. The CT report direct preference optimization device based on visual attention mask according to claim 6, characterized in that, The mask sampling module includes: An interpolation unit is used to unify the resolution and pixel spacing parameters of the various three-dimensional CT image data to obtain corresponding parameter-unified CT image data, and to extract multiple two-dimensional CT image slices from the parameter-unified CT image data at a target interval using a preset cubic spline interpolation strategy. The standardization unit is used to unify the format and character encoding of the real report text corresponding to the multiple two-dimensional CT image slices to obtain the corresponding standard report text. Based on the multiple two-dimensional CT image slices and the standard report text, the preprocessed data is determined, and the basic model is subjected to a preset supervised fine-tuning operation through the preprocessed data to obtain the basic fine-tuning model. The encoding unit is used to encode the two-dimensional CT image slices in the preprocessed data into the plurality of embedding vectors through the visual encoder of the basic fine-tuning model, and to perform random mask sampling operation on the plurality of embedding vectors based on a preset mask threshold to obtain the masked CT image.

8. An electronic device, characterized in that, include: The memory, the processor, and the computer program stored in the memory and executable on the processor, the processor executing the program to implement the CT report direct preference optimization method based on visual attention mask as described in any one of claims 1-5.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the CT report direct preference optimization method based on visual attention mask as described in any one of claims 1-5.

10. A computer program product, comprising a computer program, characterized in that, The computer program is executed to implement the CT report direct preference optimization method based on visual attention mask as described in any one of claims 1-5.

Images