Three-dimensional CT (Computed Tomography) image report generation method, system, equipment and medium

Abstract

The invention discloses a three-dimensional CT (Computed Tomography) image report generation method, system and equipment and a medium. The method comprises the following steps: acquiring a target three-dimensional CT image and a target text prompt corresponding to the target three-dimensional CT image; and inputting the target three-dimensional CT image and the target text prompt into a preset report generation model to obtain a target image report of the target three-dimensional CT image output by the report generation model, so that automatic generation from the three-dimensional CT image to the image report can be realized, the accuracy and efficiency of image report generation are improved, and the user experience is improved. And the clinical adaptability is further improved.

Description

Technical Field

[0001] The present application relates to the field of medical image processing technology, and in particular to a method, system, device and medium for generating a three-dimensional CT image report. Background Art

[0002] Medical imaging is an important part of modern medicine. It uses various imaging technologies to obtain images of the internal structure of the human body to facilitate disease diagnosis, formulation of treatment plans, and evaluation of efficacy. With the development of information technology, the application of three-dimensional medical imaging in medicine has become more and more extensive, which has not only greatly improved the accuracy and efficiency of doctors' diagnosis of diseases, but also promoted the formulation of personalized medical plans. The generation of medical imaging reports usually relies on doctors to manually interpret the images and write reports, which is a cumbersome process and easily affected by human factors.

[0003] Current three-dimensional CT image processing methods cannot fully capture the complex structural information in three-dimensional CT images. They lack high-quality, aligned multimodal data sets, making it difficult to generate reports with high clinical relevance. They cannot fully capture the global dependencies between slices and it is difficult to maintain global spatial consistency, resulting in the loss of important information in report generation and deviations between the generated content and the actual data. Summary of the invention

[0004] The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.

[0005] The main purpose of the embodiments of the present disclosure is to propose a three-dimensional CT image report generation method, system, device and storage medium, which can improve spatial consistency by enhancing the extraction of global dependencies across slices in three-dimensional CT images, thereby ensuring that the generated report is more spatially coherent, and thus making the generated report more accurate.

[0006] A first aspect of an embodiment of the present application provides a method for generating a three-dimensional CT image report for a central controller, the method comprising:

[0007] Acquire a target three-dimensional CT image and a target text prompt corresponding to the target three-dimensional CT image;

[0008] Inputting the target three-dimensional CT image and the target text prompt into a preset report generation model to obtain a target image report of the target three-dimensional CT image output by the report generation model;

[0009] The training process of the report generation model includes:

[0010] dividing the first three-dimensional CT image into a plurality of pixel blocks;

[0011] Performing linear embedding on each of the pixel blocks to obtain an initial slice of each of the pixel blocks;

[0012] Inputting each of the initial slices into a spatial coding layer to obtain a spatial feature of each of the initial slices;

[0013] Inputting the spatial features of all the initial slices into a slice coding layer to obtain a visual feature representation of the first three-dimensional CT image output by the slice coding layer;

[0014] Converting the visual feature representation of the first three-dimensional CT image into a language embedding;

[0015] Generate a text sequence according to the text prompt;

[0016] Combining the text sequence with the language embedding to obtain input features;

[0017] generating an image report of the first three-dimensional CT image according to the input feature;

[0018] evaluating the image report of the first three-dimensional CT image according to the actual report of the first three-dimensional CT image to obtain an evaluation result;

[0019] The report generation model is directly preferred optimized according to the evaluation result until a trained report generation model is obtained.

[0020] An embodiment of the present application provides a three-dimensional CT image report generation method, which obtains a target three-dimensional CT image and a target text prompt corresponding to the target three-dimensional CT image; inputs the target three-dimensional CT image and the target text prompt into a preset report generation model, and obtains a target image report of the target three-dimensional CT image output by the report generation model, which can realize automatic generation from three-dimensional CT images to image reports, improve the accuracy and efficiency of image report generation, and thereby improve clinical adaptability.

[0021] In some embodiments of the present application, before acquiring the target three-dimensional CT image and the target text prompt corresponding to the target three-dimensional CT image, the method further includes:

[0022] Acquire initial three-dimensional CT images;

[0023] The initial three-dimensional CT image is preprocessed to obtain the target three-dimensional CT image, wherein the preprocessing includes at least one of standardization, resampling, cropping, and filling.

[0024] In some embodiments of the present application, the process of generating the visual feature representation of the three-dimensional CT image by the slice coding layer includes:

[0025] Adding relative position coding to the spatial features of the plurality of initial slices to obtain a plurality of coded spatial features;

[0026] 3D average pooling is performed on the plurality of encoded spatial features to generate a visual feature representation of the three-dimensional CT image.

[0027] In some embodiments of the present application, generating an image report of the first three-dimensional CT image according to the input feature includes:

[0028] The input features are input into a 3D-CT-GPT model to obtain an image report of the first three-dimensional CT image output by the 3D-CT-GPT.

[0029] In some embodiments of the present application, the step of evaluating the image report of the first three-dimensional CT image according to the actual report of the first three-dimensional CT image to obtain the evaluation result includes:

[0030] The Chatgpt-4 evaluation method is used to evaluate the image report of the first three-dimensional CT image according to the actual report of the first three-dimensional CT image to obtain the evaluation result.

[0031] In some embodiments of the present application, directly performing preference optimization on the report generation model according to the evaluation result until a trained report generation model is obtained includes:

[0032] constructing a preference data set based on the evaluation results;

[0033] The report generation model is optimized using the preference dataset through direct preference optimization.

[0034] In some embodiments of the present application, the objective function optimized by the report generation model includes:

[0035]

[0036] D DPO ={(V,x,y w ,y l )};

[0037] in, is the objective function for optimizing the report generation model, π θ represents the model strategy, π ref represents the reference model, D DPO Represents the preferred data set, which includes the first three-dimensional CT image V, text prompt x, and positive sample y w and negative samples y l , Indicates expected value.

[0038] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present invention provides a three-dimensional CT image report generation system, the system comprising:

[0039] An acquisition module, used for acquiring a target three-dimensional CT image and a target text prompt corresponding to the target three-dimensional CT image;

[0040] A generating module, used for inputting the target three-dimensional CT image and the target text prompt into a preset report generating model, and obtaining a target image report of the target three-dimensional CT image output by the report generating model;

[0041] The training process of the report generation model includes:

[0042] dividing the first three-dimensional CT image into a plurality of pixel blocks;

[0043] Performing linear embedding on each of the pixel blocks to obtain an initial slice of each of the pixel blocks;

[0044] Inputting each of the initial slices into a spatial coding layer to obtain a spatial feature of each of the initial slices;

[0045] Inputting the spatial features of all the initial slices into a slice coding layer to obtain a visual feature representation of the first three-dimensional CT image output by the slice coding layer;

[0046] Converting the visual feature representation of the first three-dimensional CT image into a language embedding;

[0047] Generate a text sequence according to the text prompt;

[0048] Combining the text sequence with the language embedding to obtain input features;

[0049] generating an image report of the first three-dimensional CT image according to the input feature;

[0050] evaluating the image report of the first three-dimensional CT image according to the actual report of the first three-dimensional CT image to obtain an evaluation result;

[0051] The report generation model is directly preferred optimized according to the evaluation result until a trained report generation model is obtained.

[0052] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present invention provides an electronic device, comprising: at least one control processor and a memory for communicating with the at least one control processor; the memory stores instructions that can be executed by the at least one control processor, and the instructions are executed by the at least one control processor so that the at least one control processor can execute the above-mentioned three-dimensional CT image report generation method.

[0053] To achieve the above-mentioned purpose, a fourth aspect of an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the above-mentioned method for generating a three-dimensional CT image report.

[0054] It can be understood that the beneficial effects of the second to fourth aspects compared with the related art are the same as the beneficial effects of the first aspect compared with the related art. Please refer to the relevant description in the first aspect, and no further details will be given here. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0056] Figure 1 It is a flowchart of a method for generating a three-dimensional CT image report provided in an embodiment of the present application;

[0057] Figure 2 It is a model schematic diagram of an implementation method of a three-dimensional CT image report generation method provided in an embodiment of the present application;

[0058] Figure 3 It is a training schematic diagram of an implementation method of a three-dimensional CT image report generation method provided in an embodiment of the present application;

[0059] Figure 4 It is a schematic diagram of an implementation of a method for generating a three-dimensional CT image report provided in an embodiment of the present application;

[0060] Figure 5 It is a structural schematic diagram of a three-dimensional CT image report generation system provided in an embodiment of the present application;

[0061] Figure 6 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0062] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be understood as limiting the present application.

[0063] In the description of this application, if there is a description of first, second, etc., it is only for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0064] In the description of the present application, it should be understood that the descriptions involving orientation, such as the orientation or positional relationship indicated as up, down, etc., are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present application.

[0065] In the description of this application, it should be noted that, unless otherwise clearly defined, terms such as setting, installing, connecting, etc. should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meaning of the above terms in this application based on the specific content of the technical solution.

[0066] First, some nouns involved in this application are analyzed:

[0067] Noun: In the medical field, 3D CT (Computed Tomography) images refer to a series of 2D cross-sectional images acquired through computer tomography technology, which are reconstructed into 3D images after software processing. 3D CT images can provide richer and more intuitive anatomical structure information than traditional 2D CT images, helping doctors better understand the location, shape, size of the lesion and its relationship with surrounding tissues and organs.

[0068] Existing medical imaging automatic report generation technologies mainly focus on the report generation of two-dimensional (2D) images (such as X-rays and ultrasound images). However, when the existing medical imaging automatic report generation technologies are applied to three-dimensional (3D) CT images, it is difficult to achieve the same effect in a 3D environment due to the complexity of 3D image data and the need to maintain spatial consistency between multiple slices.

[0069] With the advancement of deep learning technology and the advent of the big data era, research on multimodal large language models is developing rapidly. Multimodal large language models refer to deep learning models that can process and understand multiple types of data (such as text, images, audio, etc.). Traditional language models mainly focus on text information, while multimodal models enhance their ability to understand the environment by fusing different types of inputs, thereby achieving richer and more accurate information processing and generation.

[0070] However, although the existing multimodal large language model can specifically analyze two-dimensional CT images, it lacks the ability to jointly model multimodal data and extract spatial features when processing three-dimensional CT images. It is difficult to fully capture the complex structural information in 3D CT images. Especially when generating reports, the model is prone to hallucinations, and the generated content deviates from the actual data, affecting the reliability of clinical applications.

[0071] Therefore, the current three-dimensional CT image processing methods find it difficult to maintain global spatial consistency when processing multiple slices, which affects the accuracy of diagnosis. Moreover, due to the high dimensionality and complexity of three-dimensional image data, the content generated by the existing three-dimensional CT image processing methods deviates from the actual image data, leading to clinical misjudgments. In addition, the existing three-dimensional CT image processing methods lack high-quality, aligned multimodal data sets during the training process, making it difficult to generate reports with high clinical relevance.

[0072] Based on this, the embodiments of the present application provide a three-dimensional CT image report generation method, system, electronic device and medium, aiming to realize the automatic generation of image reports from three-dimensional CT images, improve the adaptability and efficiency of image report generation, and thereby improve clinical adaptability.

[0073] The invention terms method, system, electronic device and medium provided in the embodiments of the present application are specifically explained through the following embodiments. First, the three-dimensional CT image report generation method in the embodiments of the present application is described.

[0074] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0075] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0076] The invention term method provided in the embodiment of the present application relates to the technical field. The invention term method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements a transaction resource deduction method, etc., but is not limited to the above forms.

[0077] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0078] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0079] Figure 1 This is a flow chart of a method for generating a three-dimensional CT image report provided in an embodiment of the present application. Figure 1 The method for generating a three-dimensional CT image report provided in the embodiment of the present application may include but is not limited to steps S110 to S120:

[0080] Step S110: Acquire a target three-dimensional CT image and a target text prompt corresponding to the target three-dimensional CT image.

[0081] In this step, before obtaining the target three-dimensional CT image and the target text prompt corresponding to the target three-dimensional CT image, it also includes: obtaining an initial three-dimensional CT image, and then preprocessing the initial three-dimensional CT image to obtain a target three-dimensional CT image, wherein the preprocessing includes at least one of standardization, resampling, cropping, and filling.

[0082] Preferably, the target three-dimensional CT image can be obtained by performing standardized preprocessing on the initial three-dimensional CT image, including converting the pixel values ​​of the initial CT image into standard Hounsfield Units (a standardized unit for representing tissue density in CT images), and limiting the value range to an ideal range to highlight the characteristics of lung tissue, and then unifying the spatial resolution of the initial CT image, adjusting the voxel size of the initial CT image to a fixed value by resampling, and cropping or filling the initial CT image to a uniform size, wherein the initial three-dimensional CT image can be a three-dimensional image obtained by the user by scanning and reconstructing the target area of ​​the part to be examined from multiple angles in a CT scanning room. Resampling is a technology widely used in signal processing, image processing and statistics, and is used to change the sampling rate or resolution of data, specifically including two categories: upsampling and downsampling.

[0083] Upsampling refers to increasing the sampling rate or resolution of data. In image processing, upsampling is usually used to enlarge the image to make it look clearer. In signal processing, upsampling can be used to increase the sampling frequency of the signal. Downsampling refers to reducing the sampling rate or resolution of data. In image processing, downsampling is usually used to reduce the image, reduce storage space and calculation amount. In signal processing, downsampling can be used to reduce the sampling frequency of the signal. .

[0084] In this step, the target text prompt is a report generation description of the target three-dimensional CT image, which refers to key information or parameters pre-defined for generating a report, including but not limited to: basic patient information (such as age, gender, etc.), scanning site and purpose, anatomical structures or pathological features of specific concern, or any known clinical background information.

[0085] Step S120: input the target three-dimensional CT image and the target text prompt into a preset report generation model, and obtain a target image report of the target three-dimensional CT image output by the report generation model.

[0086] In this step, the preset report generation model is the CTViT-V encoder trained by the 3D-CT-GPT model for processing three-dimensional CT image data, and then the preset report generation model is used to analyze the input target three-dimensional CT image and target text prompt to automatically generate a detailed image report.

[0087] This embodiment provides a training process in multiple iterations of the report generation model, including:

[0088] dividing the first three-dimensional CT image into a plurality of pixel blocks;

[0089] Linearly embed each pixel block to obtain the initial slice of each pixel block;

[0090] Input each initial slice into the spatial encoding layer to obtain the spatial features of each initial slice;

[0091] Inputting the spatial features of all initial slices into the slice coding layer to obtain a visual feature representation of a first three-dimensional CT image output by the slice coding layer;

[0092] Converting a visual feature representation of a first three-dimensional CT image into a language embedding;

[0093] Generate text sequences based on text prompts;

[0094] Combine text sequences with language embeddings to obtain input features;

[0095] generating an image report of the first three-dimensional CT image based on the input features;

[0096] evaluating the image report of the first three-dimensional CT image according to the actual report of the first three-dimensional CT image to obtain an evaluation result;

[0097] The report generation model is directly preferred optimized according to the evaluation results until a trained report generation model is obtained.

[0098] In this step, the first three-dimensional CT image represents the three-dimensional CT image used for training, the linear embedding is preferably a local linear embedding, and the dimension reduction is achieved by maintaining the linear relationship of the local neighborhood of the data points. The spatial encoding layer is a multi-layer spatial Transformer encoder in the CTViT-V encoder for processing each initial slice to obtain the spatial features of each initial slice. The visual feature is represented by converting the first three-dimensional CT image into a computer-processable feature vector, which is then used for various image analysis tasks such as classification, detection and recognition. Direct preference optimization is an optimization method that improves the output quality of the model by training the preference data.

[0099] In this embodiment, first, the first three-dimensional CT image is divided into multiple smaller pixel blocks, each pixel block represents a part of the first three-dimensional CT image. Specifically, the division can be performed by using a sliding window or a fixed grid, thereby reducing the computational complexity, facilitating the extraction of local features, and facilitating subsequent processing.

[0100] Then, each pixel block can be linearly embedded using linear transformation (such as fully connected layer) or convolution operation to convert it into a vector of fixed dimension. Linear transformation and convolution operation are two very important operations in deep learning. They are performed on the basis of linear algebra and are used to process and transform data to meet different task requirements.

[0101] Linear transformation specifically refers to the matrix multiplication operation, which maps a vector to another vector space. In neural networks, the most common form of linear transformation is the fully connected layer (FC layer). The fully connected layer realizes the linear combination of features by multiplying the input vector with the weight matrix and adding a bias term. The fully connected layer is mainly used to perform linear transformation on the input data to extract or convert features. In deep neural networks, the fully connected layer is usually located at the end of the network for classification tasks, that is, mapping the features extracted by the previous layer (which may be a convolutional layer or a pooling layer) to the category space.

[0102] The convolution operation is the core component of Convolutional Neural Networks (CNNs), which is mainly used to process two-dimensional data such as images. Specifically, it generates a new feature map by sliding a small filter (or kernel) on the input data for weighted summation, which can help the model capture the spatial structure information of the input data, effectively reduce the number of parameters, improve the generalization ability of the model, and keep the spatial structure of the input data unchanged.

[0103] By combining linear transformation (such as fully connected layer) or convolution operation, the high-dimensional pixel block is reduced in dimension to obtain the initial slice of each pixel block, which is used to input into the spatial coding layer to extract its spatial features, capture the spatial relationship and context information of each pixel block, and obtain the spatial features of each initial slice.

[0104] Furthermore, by inputting the spatial features of all initial slices into the slice coding layer, the spatial features of all slices are integrated to form a global understanding of the entire first three-dimensional CT image, and obtain the visual feature representation of the first three-dimensional CT image output by the slice coding layer.

[0105] Furthermore, by mapping the visual feature representation of the first three-dimensional CT image output by the slice coding layer to a space matching the embedding dimension of the 3D-GPT model subsequently used to generate evaluation results, the visual feature representation of the first three-dimensional CT image is converted into a language embedding, so that the visual feature representation can be effectively combined with the text information.

[0106] In addition, based on obtaining the text prompt representing the description of generating the first three-dimensional CT image report, a text sequence representing the text description related to the first three-dimensional CT image content is generated, and then the generated text sequence can be combined with the language embedding represented by the visual feature using splicing, weighted summation, etc. to obtain the input feature, thereby enhancing the input of the text generation model so that it can better understand the image content.

[0107] Furthermore, an image report of the first three-dimensional CT image is generated according to the input features, and then the image report of the first three-dimensional CT image and the real report of the first three-dimensional CT image are input into the 3D-GPT model to obtain evaluation results of the image report of the first three-dimensional CT image and the real report of the first three-dimensional CT image, which are used to evaluate the quality and accuracy of the generated report. Based on the evaluation results, the preferences of the generated report generated by the report generation model and the real report are compared through a direct preference optimization method, and the parameters of the report generation model are gradually adjusted to continuously improve the generation capability of the report generation model so that it can generate more accurate and professional medical imaging reports.

[0108] In some embodiments, in the step of inputting the spatial features of all initial slices into the slice coding layer to obtain the visual feature representation of the first three-dimensional CT image output by the slice coding layer, the process of the slice coding layer generating the visual feature representation of the three-dimensional CT image includes the following steps:

[0109] The relative position coding is added to the spatial features of multiple initial slices to obtain multiple coded spatial features;

[0110] 3D average pooling is performed on multiple encoded spatial features to generate visual feature representation of 3D CT images.

[0111] In this embodiment, the spatial features of all initial slices are input into the slice coding layer to obtain the visual feature representation of the first three-dimensional CT image output by the slice coding layer, which specifically refers to inputting each initial slice into the slice coding layer, adding position information to each initial slice, adding relative position coding to the spatial features of the initial slice, and then extracting its spatial features to obtain the encoded spatial features of each slice, thereby improving the ability of the report generation model to process high-resolution three-dimensional data, optimizing computing efficiency, reducing dependence on high-performance hardware, enabling the report generation model to run efficiently under existing hardware conditions, and expanding its scope of application.

[0112] Furthermore, 3D average pooling is performed on all the encoded spatial features to aggregate the features of all slices and generate a visual feature representation of the first three-dimensional CT image, thereby obtaining a compact, global visual feature representation, wherein 3D average pooling is a downsampling operation on three-dimensional data, especially for processing video data. By performing pooling in the time dimension, the number of video frames can be reduced, and by performing pooling in the spatial dimension, the resolution of each frame can be reduced, thereby reducing the amount of computation for subsequent processing, which is helpful for extracting high-level features in videos or 3D images while reducing the risk of overfitting. Feature maps can also be compressed in three dimensions (width, height, and depth, where depth usually refers to the time dimension), thereby reducing the dimensions of the data and the amount of computation for the model.

[0113] In some embodiments, the step of generating an image report of the first three-dimensional CT image according to the input feature includes the following steps:

[0114] The input features are input into the 3D-CT-GPT model to obtain an image report of the first three-dimensional CT image output by the 3D-CT-GPT.

[0115] In this embodiment, the large language model for analyzing input features is preferably a 3D-CT-GPT model, which can provide users with high-quality answers and generate text. The input features are input into a 3D-CT-GPT model that has undergone supervised fine-tuning (SFT) (i.e., a 3D-CT-GPT++ model), thereby obtaining an image report of the first three-dimensional CT image output by 3D-CT-GPT++.

[0116] In some embodiments, the step of evaluating the image report of the first three-dimensional CT image according to the actual report of the first three-dimensional CT image to obtain the evaluation result includes the following steps:

[0117] The Chatgpt-4 evaluation method is used to evaluate the image report of the first three-dimensional CT image according to the real report of the first three-dimensional CT image, and the evaluation result is obtained.

[0118] In this embodiment, the Chatgpt-4 evaluation method is used to determine the evaluation criteria for scoring and providing feedback on each generated imaging report, including the accuracy, completeness, consistency, and clinical relevance of the imaging report. Then, the real report of each three-dimensional CT imaging case, multiple imaging reports generated by the 3D-CT-GPT++ model, and the text prompt x are evaluated and analyzed. By comparing the imaging report generated by the 3D-CT-GPT++ model with the real report, the evaluation results of each imaging report are generated, including the score and detailed text feedback, which are used in the subsequent optimization process to provide data support for model optimization.

[0119] In some embodiments, the step of directly optimizing the report generation model according to the evaluation results until a trained report generation model is obtained includes the following steps:

[0120] Construct a preference dataset based on the evaluation results;

[0121] Through direct preference optimization, the report generation model is optimized using the preference dataset.

[0122] In this embodiment, by evaluating a large language model such as ChatGPT-4, a comprehensive evaluation can be performed from multiple angles, which can not only help understand the strengths and weaknesses of the model, but also ensure the reliability and security of the model in practical applications.

[0123] Furthermore, by using the scoring mechanism of Chatgpt-4 (GPT-4), the scores and detailed text feedback of the first three-dimensional CT image report and the first three-dimensional CT image real report can be obtained through the evaluation results of the image report of the first three-dimensional CT image and the real report of the first three-dimensional CT image, respectively. The image report scores higher than the threshold are taken as positive samples, and the reports with scores lower than the threshold are taken as negative samples, thereby constructing a preference data set including the first three-dimensional CT image, text prompts, positive samples and negative samples. When all the image report scores of the first three-dimensional CT image are higher or lower than the threshold, they are not included in the preference data set, and then the report generation model is optimized by using the preference data set through direct preference optimization, thereby improving the report generation efficiency and accuracy of the report generation model.

[0124] In some embodiments, in the step of optimizing the report generation model by using the preference data set through direct preference optimization, the objective function optimized by the report generation model includes:

[0125]

[0126] D DPO ={(V,x,y w ,y l )};

[0127] in, is the objective function for optimizing the report generation model, π θ represents the model strategy, π ref represents the reference model, D DPO Represents the preferred data set, which includes the first three-dimensional CT image V, text prompt x, and positive sample y w and negative samples y l , Indicates expected value.

[0128] In this embodiment, is the objective function for optimizing the report generation model, π θ Represents the model strategy, the model used to generate output in direct preference optimization, whose parameter is θ, and is optimized to generate reports that meet the preferences, π ref Denotes the reference model, a benchmark model used to compare with the policy model, and directly prefers the output of the reference model during the optimization process. DPO Represents the preferred data set, which includes the first three-dimensional CT image V, text prompt x, and positive sample y w and negative samples y l , Represents the expected value, that is, for the preference data set D DPO Calculate the average value of all samples above.

[0129] In this embodiment, the real report of the first three-dimensional CT image, the multiple image reports obtained by inputting the first three-dimensional CT image into the report generation model, and the text prompt are used as inputs of the objective function for optimizing the report generation model. The real report of the first three-dimensional CT image and the multiple image reports obtained by inputting the first three-dimensional CT image into the report generation model are compared and evaluated according to the evaluation criteria actually required, wherein the evaluation criteria include the accuracy, completeness, consistency and clinical relevance of the report, etc., so as to obtain the score and detailed text feedback of each image report, and then the report with a score higher than the threshold is used as a positive sample, and the report with a score lower than the threshold is used as a negative sample, and finally the first three-dimensional CT image V, the text prompt x, the positive sample y, and the negative sample y are obtained. w and negative samples y l Preference dataset D DPO , that is, D DPO ={(V,x,y w ,y l )}.

[0130] Furthermore, the report generation model is optimized using the preference dataset to make it more inclined to generate high-quality reports, and the strategy model (π θ ) parameters to make the reports generated by the report generation model more in line with the preferences of clinicians and improve the quality and consistency of the reports.

[0131] like Figure 2 As shown, in this embodiment, first, the pixel values ​​of the original CT image are converted to standard Hounsfield Units (HU), and the value range is limited to [-1000HU, +200HU] to highlight the characteristics of lung tissue. Then, the spatial resolution of the image is unified, the voxel size is adjusted to a fixed value (such as 0.75mm×0.75mm×1.5mm) by resampling, and the image is cropped or padded to a uniform size (such as 240×480×480) to adapt to the input requirements of the report generation model.

[0132] Furthermore, if Figure 2 In part (a), the CT image is first divided into multiple small 3D patches (pixel blocks), and each patch is linearly embedded to form an initial slice representation Z x Then, through the multi-layer spatial encoder, all the initial slices are represented by Z x Processing, capturing local structural information, extracting the spatial features of each slice The initial slice representation Z s Finally, Z s Input to the slice Transformer encoder T d In Z sBased on the slice encoder, the relative position encoding is added to model the relationship between slices (Z s The global dependency of the z-axis direction in the middle) facilitates the report generation model to understand the three-dimensional structure information and obtain the encoded feature Z v Finally, the encoded feature Z v Perform 3D average pooling to reduce the dimension of the encoded features, thereby generating a compact visual feature representation Z′ v .

[0133] Furthermore, if Figure 2 In part (b), during the vision-language fusion stage, a lightweight 2-layer MLP projection matrix W is used to represent the visual feature representation Z′. v Mapped to a space that matches the embedding dimension of the report generation model, thus forming the language embedding H v :H v =W·Z′ v .

[0134] Finally, if Figure 2 In part (b), the tokenizer of the report generation model (LLM) generates a text token sequence M based on the text prompt x (such as "Please generate a radiology report based on the following chest CT scan") q , then H v With the text token sequence M q Combined to form the complete input M of the report generation model, that is, M = concat ([M q1 ,H v ,M q2 ]), where M q1 and M q2 They are the token sequences of the text prompt x before and after the CT image placeholder.

[0135] Finally, the combined complete input M is input into the pre-trained large language model, and then the real report of CT images, that is, the medical radiology report data, is used to train and supervise the pre-trained large language model to fine-tune it, so that it has the language expression and knowledge in the medical field and can generate the image report X of CT images. a :X a =LLM(M).

[0136] Training and supervised fine-tuning of the pre-trained large language model are as follows Figure 3 As shown, in order to further improve the clinical relevance and accuracy of imaging reports, direct preference optimization technology was introduced to fine-tune the model again.

[0137] Among them, the Direct Preference Optimization (DPO) method specifically improves the performance of the model by directly optimizing the consistency between the model output and the user or decision maker's preferences. In order to achieve preference optimization, the comparison loss functions of the report generation model are determined. The comparison loss functions can measure the consistency between the model output and the user's preferences.

[0138] Further preferably, the ranking loss is used to measure whether the model correctly sorts the preference data, and then the comparison loss function is used to optimize the model parameters to maximize the matching degree between the model output and the user preference. In addition, the performance of the optimized model can also be evaluated through a test set or additional user feedback, so that the model can better adapt to the personalized needs of users.

[0139] First, if Figure 3 In part (A), the 3D-CT-GPT++ model that has been supervised fine-tuned (SFT) by setting a higher sampling temperature (such as temperature = 1.0) is used to generate multiple different image reports based on each preprocessed 3D CT medical image V and its corresponding text prompt x.

[0140] Among them, Supervised Fine-Tuning (SFT) is used to further optimize the model based on the pre-trained large-scale language model (such as BERT, GPT, etc.) through a small amount of annotated data, so that it can better adapt to specific tasks or fields. It also adjusts the parameters of the model by utilizing the existing annotated data to improve its performance on specific tasks, thereby ensuring the diversity of the generated imaging reports and capturing various reports that may be generated by the 3D-CT-GPT++ model in its current state, including high-quality and low-quality outputs, to provide sufficient data for subsequent evaluation and optimization.

[0141] Secondly, if Figure 3 In part (B), detailed evaluation requirements are determined based on actual needs, and GPT-4 is explicitly required to score and provide feedback on each generated imaging report. The evaluation criteria include the accuracy, completeness, consistency, and clinical relevance of the imaging report, and the scoring scale is usually 1 to 5 points. Then, GPT-4 will compare the imaging report generated by the 3D-CT-GPT++ model with the real report, pointing out the advantages and disadvantages, so as to obtain the score and detailed text feedback of each imaging report, and then according to the score of GPT-4, the report with a score higher than the threshold (for example, (≥3)) will be regarded as a positive sample y w , scores below the threshold are reported as negative samples y l, and then construct a model containing 3D CT medical images V, text prompts x, and positive samples y w and negative samples y l The preference data set, namely D dPO ={(V,x,y w ,y l )}.

[0142] If all the imaging reports of a 3D CT imaging case have scores above or below the threshold, it will not be included in the preference dataset. Figure 3 In part (C), the parameters of the 3D-CT-GPT++ model are updated according to the objective function of the direct preference optimization method. The specific objective function is as follows:

[0143]

[0144] D DPo ={(V,x,y w ,y l )};

[0145] in, is the objective function for optimizing the report generation model, π θ Represents the model strategy, the model used to generate output in direct preference optimization, whose parameter is θ, and is optimized to generate reports that meet the preferences, π ref Denotes the reference model, a benchmark model used to compare with the policy model, and directly prefers the output of the reference model during the optimization process. DPO Represents the preferred data set, which includes the first three-dimensional CT image V, text prompt x, and positive sample y w and negative samples y l , Represents the expected value, that is, for the preference data set D DPO Calculate the average value of all samples above.

[0146] Finally, the optimized 3D-CT-GPT++ model was obtained, which can automatically generate accurate, coherent and clinically compliant imaging reports from preprocessed 3D medical images.

[0147] In addition, in one implementation of this embodiment, the training of the 3D-CT-GPT++ model is divided into four main stages, specifically including:

[0148] The image encoder training phase involves training the CTViT-V encoder on a large-scale 3D CT scan dataset using the CT-CLIP framework. Figure 3(C) As shown in the image encoder training phase, the CT-CLIP framework is used to train the CTViT-V encoder. CT-CLIP is a 3D adaptation of the CLIP architecture designed for self-supervised contrastive learning. Through contrastive learning, 3D CT images and their corresponding radiology reports are aligned in a shared latent space, thereby learning effective and discriminative image representations. Specifically, by maximizing the cosine similarity between matching 3D CT scans and corresponding radiology reports (positive pairs) and minimizing the similarity between non-matching pairs (negative pairs), the CTViT-V encoder is able to extract highly discriminative features from 3D images and project them into the same 512-dimensional space as the text encoder.

[0149] In the pre-training phase, the 3D-CT-GPT++ model adopts an interactive approach across multiple data types, and is trained to understand the relationship between 3D CT image features and their corresponding reports by analyzing a large number of 3DCT image-report pairs.

[0150] Fine-tuning phase: This includes optimizing the 3D-CT-GPT++ model and aligning 3D CT image features with specific imaging reports. There are two fine-tuning strategies:

[0151] Lightweight fine-tuning based on LoRA: Lightweight fine-tuning is performed using LoRA (low rank adaptation). The image encoder remains frozen, and some parameters of the language model and projection layer are fine-tuned to help avoid overfitting.

[0152] Supervised Fine-tuning (SFT): By applying supervised fine-tuning (SFT), all parameters of the language model are optimized using labeled data, so that the number of training rounds is carefully chosen through cross-validation to prevent overfitting.

[0153] The direct preference optimization training phase involves performing direct preference optimization (DPO) of the 3D-CT-GPT++ model after supervised fine-tuning of the 3D-CT-GPT++ model to ensure that the generated reports are closer to the clinicians' diagnostic preferences.

[0154] Therefore, the 3D-CT-GPT++ model uses the improved CTViT-V encoder to efficiently extract 3D image features and the pre-trained large language model (LLM) to accurately generate reports, realizing the automatic generation of radiology reports from 3D CT images, significantly improving the efficiency and accuracy of image report generation. In addition, using direct preference optimization technology, GPT-4 is used to score and feedback the generated reports, build a preference data set, and optimize the strategy model, making it more inclined to generate high-quality reports that meet clinical needs, improving the stability and effect of training.

[0155] like Figure 4As shown, in this embodiment, taking a patient's chest 3D CT scan as an example, the original 3D CT image of the patient is firstly subjected to standardized preprocessing, converted into Hounsfield Units (HU), and the value range is limited to the range of [-1000HU, +200HU] to highlight the characteristics of lung tissue. Subsequently, the voxel size is adjusted to a fixed 0.75mm×0.75mm×1.5mm by resampling, and the image is uniformly sized to 240×480×480 by image processing such as cropping or padding.

[0156] Furthermore, the preprocessed 3D CT images are input into the improved CTViT-V encoder for feature extraction to obtain visual feature representation. Then, through the visual-language fusion stage, a two-layer multi-layer perceptron (MLP) projection matrix is ​​used to map the visual features into language embeddings, and combined with the text prompt "Please generate a radiology report based on the following chest CT scan" to form the complete input M through splicing.

[0157] Finally, the combined input M is input into the 3D-CT-GPT++ model after direct preference optimization and supervised fine-tuning to obtain the image report of the three-dimensional CT image.

[0158] In this embodiment, by obtaining a target three-dimensional CT image and a target text prompt corresponding to the target three-dimensional CT image; inputting the target three-dimensional CT image and the target text prompt into a preset report generation model, and obtaining a target image report of the target three-dimensional CT image output by the report generation model, it is possible to realize automatic generation from three-dimensional CT images to image reports, thereby improving the accuracy and efficiency of image report generation, and thereby improving clinical adaptability.

[0159] like Figure 5 As shown, some embodiments of the present application provide a three-dimensional CT image report generation system, the system includes an acquisition module 510 and a generation module 520, specifically:

[0160] The acquisition module 510 is used to acquire a target three-dimensional CT image and a target text prompt corresponding to the target three-dimensional CT image.

[0161] A generating module 520, for inputting a target three-dimensional CT image and a target text prompt into a preset report generating model, and obtaining a target image report of the target three-dimensional CT image output by the report generating model;

[0162] The training process of the report generation model includes:

[0163] dividing the first three-dimensional CT image into a plurality of pixel blocks;

[0164] Linearly embed each pixel block to obtain the initial slice of each pixel block;

[0165] Input each initial slice into the spatial encoding layer to obtain the spatial features of each initial slice;

[0166] Inputting the spatial features of all initial slices into the slice coding layer to obtain a visual feature representation of a first three-dimensional CT image output by the slice coding layer;

[0167] Converting a visual feature representation of a first three-dimensional CT image into a language embedding;

[0168] Generate text sequences based on text prompts;

[0169] Combine text sequences with language embeddings to obtain input features;

[0170] generating an image report of the first three-dimensional CT image based on the input features;

[0171] evaluating the image report of the first three-dimensional CT image according to the actual report of the first three-dimensional CT image to obtain an evaluation result;

[0172] The report generation model is directly preferred optimized according to the evaluation results until a trained report generation model is obtained.

[0173] In some implementations, the acquisition module 510 may include: acquiring an initial three-dimensional CT image.

[0174] In some implementations, the acquisition module 510 may include: preprocessing the initial three-dimensional CT image to obtain a target three-dimensional CT image, wherein the preprocessing includes at least one of standardization, resampling, cropping, and filling.

[0175] In some implementations, the generation module 520 may include: adding relative position coding to the spatial features of multiple initial slices to obtain multiple encoded spatial features.

[0176] In some implementations, the generation module 520 may include: performing 3D average pooling on a plurality of encoded spatial features to generate a visual feature representation of a three-dimensional CT image.

[0177] In some embodiments, the generation module 520 may include: inputting the input features into the 3D-CT-GPT model to obtain an image report of the first three-dimensional CT image output by the 3D-CT-GPT.

[0178] In some implementations, the generation module 520 may include: using the Chatgpt-4 evaluation method to evaluate the image report of the first three-dimensional CT image according to the actual report of the first three-dimensional CT image to obtain an evaluation result.

[0179] In some implementations, the generating module 520 may include: constructing a preference data set according to the evaluation results.

[0180] In some implementations, the generation module 520 may include optimizing the report generation model using the preference data set through direct preference optimization.

[0181] In some implementations, the generation module 520 may include:

[0182]

[0183] D DPO ={(V,x,y w ,y l )};

[0184] in, is the objective function for optimizing the report generation model, π θ represents the model strategy, π ref represents the reference model, D DPO Represents the preferred data set, which includes the first three-dimensional CT image V, text prompt x, and positive sample y w and negative samples y l , Indicates expected value.

[0185] It should be noted that the three-dimensional CT image report generation system provided in this embodiment and the above-mentioned three-dimensional CT image report generation method are based on the same inventive concept, so the relevant content of the above-mentioned three-dimensional CT image report generation method is also applicable to the content of the three-dimensional CT image report generation system, and therefore, it will not be repeated here.

[0186] In order to improve the efficiency and accuracy of generating three-dimensional CT image reports, the system obtains the target three-dimensional CT image and the target text prompt corresponding to the target three-dimensional CT image; the target three-dimensional CT image and the target text prompt are input into a preset report generation model, and the target image report of the target three-dimensional CT image is obtained as output by the report generation model. The system can realize the automatic generation from three-dimensional CT images to image reports, improve the accuracy and efficiency of image report generation, and thus improve clinical adaptability.

[0187] An embodiment of the present application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor implements the above-mentioned three-dimensional CT image report generation method when executing the computer program.

[0188] like Figure 6 , Figure 6 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application, wherein the electronic device includes:

[0189] At least one battery;

[0190] at least one memory;

[0191] at least one processor;

[0192] at least one program;

[0193] The program is stored in the memory, and the processor executes at least one program to implement the above-mentioned three-dimensional CT image report generation method implemented in the present disclosure.

[0194] The electronic device may be any intelligent terminal including a mobile phone, a tablet computer, a personal digital assistant (PDA), a vehicle-mounted computer, etc.

[0195] The electronic device according to the embodiment of the present application is described in detail below.

[0196] The processor 1600 may be implemented by a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present disclosure;

[0197] The memory 1700 may be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1700 may store an operating system and other application programs. When the technical solution provided in the embodiments of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1700, and the processor 1600 calls and executes a method for generating a three-dimensional CT image report in the embodiments of the present disclosure.

[0198] Input / output interface 1800, used to implement information input and output;

[0199] The communication interface 1900 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0200] Bus 2000 , which transmits information between various components of the device (e.g., processor 1600 , memory 1700 , input / output interface 1800 , and communication interface 1900 );

[0201] The processor 1600 , the memory 1700 , the input / output interface 1800 , and the communication interface 1900 are connected to each other in communication within the device via the bus 2000 .

[0202] The embodiment of the present disclosure also provides a storage medium, which is a computer-readable storage medium. The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the above-mentioned three-dimensional CT image report generation method.

[0203] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0204] The embodiments described in the embodiments of the present disclosure are intended to more clearly illustrate the technical solutions of the embodiments of the present disclosure and do not constitute a limitation on the technical solutions provided by the embodiments of the present disclosure. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present disclosure are also applicable to similar technical problems.

[0205] Those skilled in the art will appreciate that the technical solutions shown in the figures do not limit the embodiments of the present disclosure and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0206] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0207] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0208] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0209] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0210] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0211] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0212] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0213] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable an electronic device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of each embodiment method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store programs.

[0214] The above is a specific description of the preferred implementation of the embodiments of the present application, but the embodiments of the present application are not limited to the above-mentioned implementation methods. Technical personnel familiar with the field can also make various equivalent modifications or substitutions without violating the spirit of the embodiments of the present application. These equivalent modifications or substitutions are all included in the scope defined by the claims of the embodiments of the present application.

[0215] The embodiments of the present application are described in detail above in conjunction with the accompanying drawings, but the present application is not limited to the above embodiments. Various changes can be made within the knowledge scope of ordinary technicians in the relevant technical field without departing from the purpose of the present application.

Claims

1. A method for generating a three-dimensional CT image report, characterized in that: The method comprises: Acquire a target three-dimensional CT image and a target text prompt corresponding to the target three-dimensional CT image; Inputting the target three-dimensional CT image and the target text prompt into a preset report generation model to obtain a target image report of the target three-dimensional CT image output by the report generation model; The training process of the report generation model includes: dividing the first three-dimensional CT image into a plurality of pixel blocks; Performing linear embedding on each of the pixel blocks to obtain an initial slice of each of the pixel blocks; Inputting each of the initial slices into a spatial coding layer to obtain a spatial feature of each of the initial slices; Inputting the spatial features of all the initial slices into a slice coding layer to obtain a visual feature representation of the first three-dimensional CT image output by the slice coding layer; Converting the visual feature representation of the first three-dimensional CT image into a language embedding; Generate a text sequence according to the text prompt; Combining the text sequence with the language embedding to obtain input features; generating an image report of the first three-dimensional CT image according to the input feature; evaluating the image report of the first three-dimensional CT image according to the actual report of the first three-dimensional CT image to obtain an evaluation result; The report generation model is directly preferred optimized according to the evaluation result until a trained report generation model is obtained.

2. The method for generating a three-dimensional CT image report according to claim 1, characterized in that: Before acquiring the target three-dimensional CT image and the target text prompt corresponding to the target three-dimensional CT image, the method further includes: Acquire initial three-dimensional CT images; The initial three-dimensional CT image is preprocessed to obtain the target three-dimensional CT image, wherein the preprocessing includes at least one of standardization, resampling, cropping, and filling.

3. The method for generating a three-dimensional CT image report according to claim 1, characterized in that: The process of generating the visual feature representation of the three-dimensional CT image by the slice coding layer includes: Adding relative position coding to the spatial features of the plurality of initial slices to obtain a plurality of coded spatial features; 3D average pooling is performed on the plurality of encoded spatial features to generate a visual feature representation of the three-dimensional CT image.

4. The method for generating a three-dimensional CT image report according to claim 1, characterized in that: Generating an image report of the first three-dimensional CT image according to the input feature includes: The input features are input into a 3D-CT-GPT model to obtain an image report of the first three-dimensional CT image output by the 3D-CT-GPT.

5. The method for generating a three-dimensional CT image report according to claim 1, characterized in that: The step of evaluating the image report of the first three-dimensional CT image according to the true report of the first three-dimensional CT image to obtain an evaluation result includes: The Chatgpt-4 evaluation method is used to evaluate the image report of the first three-dimensional CT image according to the actual report of the first three-dimensional CT image to obtain the evaluation result.

6. The method for generating a three-dimensional CT image report according to claim 1, characterized in that: The directly performing preference optimization on the report generation model according to the evaluation result until a trained report generation model is obtained comprises: constructing a preference data set based on the evaluation results; The report generation model is optimized using the preference dataset through direct preference optimization.

7. The method for generating a three-dimensional CT image report according to claim 6, characterized in that: The objective function optimized by the report generation model includes: D DPO ={(V,x,y w ,y l )}; in, is the objective function for optimizing the report generation model, π θ represents the model strategy, π ref represents the reference model, D DPO Represents the preferred data set, which includes the first three-dimensional CT image V, text prompt x, and positive sample y w and negative samples y l , Indicates expected value.

8. A three-dimensional CT image report generation system, characterized in that: The system comprises: An acquisition module, used for acquiring a target three-dimensional CT image and a target text prompt corresponding to the target three-dimensional CT image; A generating module, used for inputting the target three-dimensional CT image and the target text prompt into a preset report generating model, and obtaining a target image report of the target three-dimensional CT image output by the report generating model; The training process of the report generation model includes: dividing the first three-dimensional CT image into a plurality of pixel blocks; Performing linear embedding on each of the pixel blocks to obtain an initial slice of each of the pixel blocks; Inputting each of the initial slices into a spatial coding layer to obtain a spatial feature of each of the initial slices; Inputting the spatial features of all the initial slices into a slice coding layer to obtain a visual feature representation of the first three-dimensional CT image output by the slice coding layer; Converting the visual feature representation of the first three-dimensional CT image into a language embedding; Generate a text sequence according to the text prompt; Combining the text sequence with the language embedding to obtain input features; generating an image report of the first three-dimensional CT image according to the input feature; evaluating the image report of the first three-dimensional CT image according to the actual report of the first three-dimensional CT image to obtain an evaluation result; The report generation model is directly preferred optimized according to the evaluation result until a trained report generation model is obtained.

9. An electronic device, characterized in that: It includes at least one control processor and a memory for communicating with the at least one control processor; the memory stores instructions that can be executed by the at least one control processor, and the instructions are executed by the at least one control processor so that the at least one control processor can execute a three-dimensional CT image report generation method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the three-dimensional CT image report generation method according to any one of claims 1 to 7.

Images