A method and system for generating a diagnostic report

Abstract

The application relates to the medical technology field and provides a method and system for generating a diagnosis report.The method comprises the following steps: obtaining scan raw data, a reconstructed image and clinical information of a target object; determining first lesion information and second lesion information of the target object based on the scan raw data and the reconstructed image; determining whether the target object contains a lesion based on the first lesion information and the second lesion information; in response to determining that a lesion region is contained, determining a target reconstructed image of the lesion region by using a simulation scanning model; performing feature extraction on the target reconstructed image to obtain third lesion information; and generating a structured diagnosis report by using a report generation model based on the first lesion information, the second lesion information, the third lesion information and the clinical information.The application generates a structured diagnosis report based on the first lesion information, the second lesion information, the third lesion information and the clinical information, the information is complementary and verified with each other, the diagnosis accuracy is significantly improved, and the risk of missed diagnosis and misdiagnosis caused by single information is reduced.

Description

TECHNICAL FIELD

[0001] The present specification relates to the technical field of medical technology, and in particular, to a method and system for generating a diagnosis report. BACKGROUND

[0002] The current routine process for diagnosing a patient is as follows: a doctor issues an examination order, and the patient undergoes scanning; then, a reconstructed medical image is obtained using the scan raw data, and the reconstructed medical image is successively judged by an imaging doctor and a clinical doctor to determine whether additional scanning (e.g., high-resolution scanning, enhanced scanning, etc.) is needed for the lesion, and if it is determined that additional scanning for the lesion is needed, the clinical doctor issues an examination order again, and additional scanning is performed. This is a serial and artificial process, which leads to a long diagnosis and treatment cycle and low efficiency.

[0003] Secondly, the clinical doctor mainly relies on a single reconstructed image for diagnosis, and the information dimension is limited, which may lead to an incomplete understanding of the lesion, and thus cause two risks: one is missed diagnosis or misdiagnosis, and the other is defensive and unnecessary additional scanning for clear diagnosis, which increases the radiation exposure, economic cost and time cost of the patient.

[0004] Therefore, it is desirable to provide an efficient and accurate method and system for generating a diagnosis report. SUMMARY

[0005] One of the embodiments of the present specification provides a method for generating a diagnosis report. The method comprises: obtaining scan raw data of a target object, a reconstructed image of the target object and clinical information of the target object, the scan raw data being obtained by scanning the target object through a medical imaging device, and the reconstructed image being generated based on the scan raw data; determining first lesion information of the target object based on the scan raw data; determining second lesion information of the target object based on the reconstructed image and the first lesion information; determining whether the target object contains a lesion based on the first lesion information and the second lesion information; in response to determining that the target object contains a lesion region, determining a target reconstructed image of the lesion region based on the scan raw data and the reconstructed image using a simulation scanning model; performing feature extraction on the target reconstructed image to obtain third lesion information; and generating a structured diagnosis report of the target object based on the first lesion information, the second lesion information, the third lesion information and the clinical information using a report generation model.

[0006] One of the embodiments of the present specification provides a system for generating a diagnostic report. The system comprises an acquisition module, a determination module, a simulated scanning module and a report generation module. The acquisition module is configured to acquire scan raw data of a target object, a reconstructed image of the target object and clinical information of the target object, the scan raw data being obtained by scanning the target object by a medical imaging device, and the reconstructed image being generated based on the scan raw data. The determination module is configured to: determine first lesion information of the target object based on the scan raw data; determine second lesion information of the target object based on the reconstructed image and the first lesion information; and determine whether the target object contains a lesion based on the first lesion information and the second lesion information. The simulated scanning module is configured to, in response to determining that the target object contains a lesion region, determine a target reconstructed image of the lesion region based on the scan raw data and the reconstructed image using a simulated scanning model; and the determination module is further configured to perform feature extraction on the target reconstructed image to obtain third lesion information. The report generation module is configured to generate a structured diagnostic report of the target object based on the first lesion information, the second lesion information, the third lesion information and the clinical information using a report generation model.

[0007] One of the embodiments of the present specification provides a system for determining a diagnostic report. The system comprises at least one storage device for storing computer instructions; and at least one processor for executing the computer instructions to implement the method for determining the blood flow reserve fraction.

[0008] According to some embodiments of the present application, the structured diagnostic report is generated based on the first lesion information, the second lesion information, the third lesion information and the clinical information at the same time, which can complement and verify each other, significantly improve the robustness and accuracy of diagnosis, and reduce the risk of missed diagnosis and misdiagnosis caused by single information.

[0009] Some additional features of the present application can be described in the following description. Some additional features of the present application will become apparent to those skilled in the art from the following description and corresponding drawings, or from the practice of the embodiments. The features of the present application can be realized and obtained by practicing or using the methods, means and combinations set forth in the detailed examples described below. BRIEF DESCRIPTION OF DRAWINGS

[0010] The present specification will be further illustrated in the form of exemplary embodiments, which will be described in detail with reference to the accompanying drawings. These embodiments are not limiting, and in these embodiments, the same numbers represent the same structures, wherein:

[0011] Figure 1 is a schematic diagram of an application scenario of an exemplary report generation system according to some embodiments of the present specification;

[0012] Figure 2is a schematic diagram of an exemplary report generation system according to some embodiments of the present specification;

[0013] Figure 3 is a schematic diagram of a flow of exemplary generating a structured diagnostic report according to some embodiments of the present specification;

[0014] Figure 4 is a schematic diagram of an exemplary training process of a data analysis model and an image analysis model according to some embodiments of the present specification;

[0015] Figure 5 is a schematic diagram of an exemplary process of determining a first weight, a second weight and a third weight according to some embodiments of the present specification;

[0016] Figure 6 is a schematic diagram of an exemplary process of determining a structured diagnostic report using a report generation model according to some embodiments of the present specification;

[0017] Figure 7 is a schematic diagram of an exemplary process of generating a structured diagnostic report according to some embodiments of the present specification. DETAILED DESCRIPTION

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present specification, the following will briefly introduce the drawings needed to be used in the embodiments description. Obviously, the drawings in the following description are only some examples or embodiments of the present specification, and for those skilled in the art, the present specification can also be applied to other similar scenarios without creative labor on the basis of these drawings. Unless it is clear from the language context or otherwise stated, the same reference numbers in the drawings represent the same structure or operation.

[0019] It should be understood that the "system", "device", "unit" and / or "module" used herein is a method for distinguishing different components, elements, parts, portions or assemblies at different levels. However, if other words can achieve the same purpose, the words can be replaced by other expressions.

[0020] As shown in the specification and claims, unless the context clearly indicates otherwise, "one", "a", "an" and / or "the" do not refer to the singular, but can also include the plural. Generally speaking, the terms "comprise" and "include" only indicate the inclusion of the steps and elements explicitly identified, and these steps and elements do not constitute an exclusive list, and the method or device can also include other steps or elements.

[0021] Flow diagrams are used in this specification to illustrate the operations according to embodiments of the present specification in terms of a system performing actions in a particular order. It will be understood that the actions can be performed at any order by various embodiments, and the ordering of actions can be changed. It will also be understood that additional actions can be added or removed from the processes, or one or more actions can be performed concurrently. Furthermore, some actions can be performed by different components or in different components than those described in this specification.

[0022] Figure 1 FIG. 1 is a schematic diagram of an application scenario of an exemplary report generation system according to some embodiments of the present specification. As shown in FIG. 1, the application scenario 100 of the report generation system can include a medical imaging device 110, a processing device 120, a terminal device 130, a storage device 140, and a network 150. In some embodiments, the processing device 120 can be part of the medical imaging device 110. The connections between the components in the application scenario 100 can be variable. For example, the medical imaging device 110 can be connected to the processing device 120 through the network 150. As another example, the medical imaging device 110 can be directly connected to the processing device 120. As a further example, the storage device 140 can be directly or through the network 150 connected to the processing device 120. As yet another example, the terminal device 130 can be directly connected to the processing device 120 (as shown by the dashed arrow connecting the terminal device 130 and the processing device 120) or through the network 150 connected to the processing device 120. Figure 1 Figure 1

[0023] ​​The medical imaging device 110 can be a non-invasive scanning imaging device for disease diagnosis or research purposes. In some embodiments, the medical imaging device 110 can scan an object within a detection area or a scanning area to obtain scan raw data of the object. In some embodiments, the medical imaging device 110 can include a single modality scanner and / or a multi-modality scanner. The single modality scanner can include, for example, an ultrasound scanner, an X-ray scanner, a computed tomography (CT) scanner, a magnetic resonance imaging (MRI) scanner, a positron emission tomography (PET) scanner, an optical coherence tomography (OCT) scanner, an ultrasound (US) scanner, an intravascular ultrasound (IVUS) scanner, or the like, or any combination thereof. The multi-modality scanner can include, for example, an X-ray imaging-magnetic resonance imaging (X-ray-MRI) scanner, a positron emission tomography-X-ray imaging (PET-X-ray) scanner, a single photon emission computed tomography-magnetic resonance imaging (SPECT-MRI) scanner, a positron emission tomography-computed tomography (PET-CT) scanner, a digital subtraction angiography-magnetic resonance imaging (DSA-MRI) scanner, or the like. In some embodiments, the processing device 120 can be integrated on the medical imaging device 110, or the medical imaging device 110 and the processing device 120 can implement their functions by the same entity. The medical imaging devices provided above are for illustrative purposes only and are not intended to limit the scope of the present specification.

[0024] The processing device 120 can process data and / or information obtained from the medical imaging device 110, the terminal device 130, the storage device 140, or other components of the application scenario 100. For example, the processing device 120 obtains scan raw data of a target object, and reconstructs a reconstructed image of the target object based on the scan raw data. The processing device 120 determines first lesion information of the target object based on the scan raw data, and then determines second lesion information of the target object based on the reconstructed image and the first lesion information. Further, the processing device 120 determines whether the target object contains a lesion based on the first lesion information and the second lesion information. In response to determining that the target object contains a lesion area, the processing device 120 determines a target reconstructed image of the lesion area using a simulated scanning model based on the scan raw data and the reconstructed image, and performs feature extraction on the target reconstructed image to obtain third lesion information. Then, the processing device 120 generates a structured diagnostic report of the target object based on the first lesion information, the second lesion information, the third lesion information, and clinical information using a report generation model.

[0025] In some embodiments, the processing device 120 can be local or remote. For example, the processing device 120 can access information and / or data from the medical imaging device 110, the terminal device 130, and / or the storage device 140 through the network 150.

[0026] The terminal device 130 can include a mobile device 131, a tablet 132, a notebook 133, or the like, or any combination thereof. In some embodiments, the terminal device 130 can be a part of the processing device 120.

[0027] The storage device 140 can store data, instructions, and / or any other information. In some embodiments, the storage device 140 can store data obtained from the medical imaging device 110, the processing device 120, and / or the terminal device 130, such as scan raw data acquired by the medical imaging device 110, and the like.

[0028] The network 150 can include any suitable network capable of facilitating exchange of information and / or data. In some embodiments, at least one component of the application scenario 100 (e.g., the medical imaging device 110, the processing device 120, the terminal device 130, the storage device 140) can exchange information and / or data with at least one other component of the application scenario 100 through the network 150. For example, the processing device 120 can obtain scan raw data of a target object from the medical imaging device 110 through the network 150. For another example, the terminal device 130 can obtain a structured diagnosis report of the target object from the processing device 120 through the network 150.

[0029] It should be noted that the application scenario 100 is provided for illustrative purposes only and is not intended to limit the scope of the present specification. Various modifications or changes can be made according to the description of the present specification by those of ordinary skill in the art. For example, the application scenario 100 can also include input devices and / or output devices. For another example, the application scenario 100 can implement similar or different functions on other devices. However, these changes and modifications will not depart from the scope of the present specification.

[0030] Figure 2 is a schematic diagram of an exemplary report generation system according to some embodiments of the present specification.

[0031] As shown in Figure 2 In some embodiments, the report generation system 200 can include an obtaining module 210, a determining module 220, a simulated scanning module 230, and a report generation module 240. In some embodiments, the report generation system 200 can further include a model training module 250. In some embodiments, the functions of the report generation system 200 can be performed by the processing device 120, for example, the obtaining module 210, the determining module 220, the simulated scanning module 230, the report generation module 240, and the model training module 250 can be modules in the processing device 120.

[0032] The acquisition module 210 can be configured to acquire scan raw data of the target object, a reconstructed image of the target object, and clinical information of the target object. The scan raw data is obtained by scanning the target object through a medical imaging device, and the reconstructed image is generated based on the scan raw data. For more description of acquiring the scan raw data of the target object, the reconstructed image of the target object, and the clinical information of the target object, please refer to step 310 in the method 300. Figure 3

[0033] The determination module 220 can be configured to determine first lesion information of the target object based on the scan raw data. For more description of determining the first lesion information of the target object based on the scan raw data, please refer to step 320 in the method 300. Figure 3

[0034] The determination module 220 can also be configured to determine second lesion information of the target object based on the reconstructed image and the first lesion information. For more description of determining the second lesion information of the target object based on the reconstructed image and the first lesion information, please refer to step 330 in the method 300. Figure 3

[0035] The determination module 220 can further be configured to determine whether the target object contains a lesion based on the first lesion information and the second lesion information. For more description of determining whether the target object contains a lesion based on the first lesion information and the second lesion information, please refer to step 340 in the method 300. Figure 3

[0036] The simulation scanning module 230 can be configured to, in response to determining that the target object contains a lesion region, determine a target reconstructed image of the lesion region based on the scan raw data and the reconstructed image using a simulation scanning model. For more description of determining the target reconstructed image of the lesion region based on the scan raw data and the reconstructed image using the simulation scanning model, please refer to step 350 in the method 300. Figure 3

[0037] The determination module 220 can be configured to perform feature extraction on the target reconstructed image to obtain third lesion information. For more description of performing feature extraction on the target reconstructed image to obtain the third lesion information, please refer to step 360 in the method 300. Figure 3

[0038] The report generation module 240 can be configured to generate a structured diagnostic report of the target object based on the first lesion information, the second lesion information, the third lesion information, and the clinical information using a report generation model. For more description of generating the structured diagnostic report of the target object using the report generation model, please refer to step 370 in the method 300. Figure 3

[0039] It should be understood that, Figure 2 ​​​​​​​The illustrated system and its modules can be implemented in various ways. For example, in some embodiments the system and its modules can be implemented by hardware, software, or a combination of hardware and software.

[0040] It should be noted that the above description of the system and its modules is for convenience of description only, and is not intended to limit the scope of the present specification to the illustrated embodiments. It will be appreciated by those skilled in the art that, after understanding the principles of the system, various modules can be combined in any manner, or connected with other modules to form a subsystem, without departing from the principles of the system. For example, in some embodiments, Figure 2 The above modules disclosed in the present specification can be different modules in a system, or a module can implement the functions of two or more modules described above. For example, each module can share a storage module, and each module can also have its own storage module. In some embodiments, the model training module 250 and other modules can be implemented by different systems. For example, the model training module 250 can be implemented by a computing device of a provider of a machine learning model, and the other modules can be implemented by a computing device of a user of the machine learning model. Variations such as this are within the scope of protection of the present specification.

[0041] Figure 3 is a flowchart illustrating an example process of generating a structured diagnostic report according to some embodiments of the present specification. In some embodiments, one or more steps of the flow 300 can be performed by the modules of the processing device 120. Figure 1 The application scenario 100 illustrated can be implemented or executed by the report generation system 200 illustrated. Figure 2 For example, the flow 300 can be executed by the modules of the processing device 120. As illustrated, Figure 3 The flow 300 can include the following steps.

[0042] Step 310: Obtain scan raw data of a target object, a reconstructed image of the target object, and clinical information of the target object, the scan raw data being obtained by scanning the target object by a medical imaging device, and the reconstructed image being generated based on the scan raw data. In some embodiments, step 310 can be executed by the processing device 120 or the obtaining module 210.

[0043] The target object refers to an object that needs to undergo medical scanning. The target object can include the whole or part of a biological object and / or a non-biological object involved in the scanning process. The following is described by taking a human body as an example.

[0044] Scan-raw data is raw data obtained by scanning a target object using a medical imaging device 110. For example, when a target object is scanned using an MRI scanner, the scan-raw data can include k-space data. For another example, when a target object is scanned using a CT scanner, the scan-raw data can include projection data or sinogram data. For yet another example, when a target object is scanned using a PET scanner, the scan-raw data includes coincidence event data. In some embodiments, the processing device 120 can obtain the scan-raw data directly from the medical imaging device 110. In some embodiments, the scan-raw data can be generated in advance and stored in a storage device (e.g., the storage device 140 or an external storage device), from which the processing device 120 can obtain the scan-raw data.

[0045] A reconstructed image refers to a medical image containing the anatomical structure of a target object. Exemplary reconstructed images include CT images, MRI images, PET images, etc. In some embodiments, the processing device 120 can reconstruct the scan-raw data using various reconstruction algorithms to generate a reconstructed image. For example, for CT scanning, exemplary reconstruction algorithms include analytical reconstruction algorithms, iterative reconstruction algorithms, etc. For another example, for MRI scanning, exemplary reconstruction algorithms include two / three-dimensional Fourier transform algorithms, compressed sensing reconstruction algorithms, deep learning or artificial intelligence based reconstruction algorithms, etc. For yet another example, for PET scanning, exemplary reconstruction algorithms include analytical reconstruction algorithms (e.g., Filtered Back Projection (FBP), iterative reconstruction algorithms, etc. Iterative reconstruction algorithms can include Maximum-Likelihood Expectation-Maximization (MLEM), iterative reconstruction algorithms based on point spread function modeling, deep learning based PET reconstruction algorithms, etc.

[0046] In some embodiments, the processing device 120 can generate a reconstructed image based on the scan-raw data and the confidence corresponding to the scan-raw data (e.g., the first confidence map in step 320). For regions with high confidence, the reconstructed image is retained and enhanced. For regions with low confidence (e.g., noise, artifacts), the reconstructed image is suppressed or interpolated, thereby significantly improving the quality of the reconstructed image.

[0047] The clinical information of a target object includes basic information of the target object, current symptom information, medical history, doctor's preliminary diagnosis result, etc. The basic information includes age, gender, height, weight, occupation and living environment, etc. The medical history includes historical diagnosis disease information, historical surgery information, historical trauma information, etc. The processing device 120 can obtain the clinical information of the target object from the medical information system of a hospital.

[0048] At step 320, first lesion information of the target object is determined based on the scan raw data. In some embodiments, step 320 can be performed by the processing device 120 or the determining module 220.

[0049] The first lesion information refers to information related to a lesion of the target object, which is determined based on the scan raw data. The first lesion information includes a first lesion detection result. The first lesion detection result can include whether there is a lesion, a number of lesions, a severity of a lesion, a type of a lesion (e.g., a solid tumor, a cyst, a hemorrhage, etc.). The severity of a lesion refers to a probability that the lesion is a benign lesion or a malignant lesion. The lower the probability that the lesion is a benign lesion, the higher the severity of the lesion; the higher the probability that the lesion is a benign lesion, the lower the severity of the lesion. The lower the probability that the lesion is a malignant lesion, the lower the severity of the lesion; the higher the probability that the lesion is a malignant lesion, the higher the severity of the lesion. In some embodiments, the first lesion detection result further includes a size of a lesion, a location of a lesion. For example, the scan raw data is projection data, and the first lesion detection result includes a location of a lesion in a projection domain.

[0050] In some embodiments, the first lesion information further includes a first confidence map. The first confidence map contains a confidence of each raw data point in the scan raw data. The confidence of each raw data point is between 0 and 1. The higher the confidence score of a raw data point, the higher the confidence of the raw data point, i.e., the higher the possibility that the signal of the data point is derived from the anatomical structure of the target object, and the lower the possibility that the signal of the data point is derived from noise or artifact.

[0051] In some embodiments, the first lesion information is determined by processing the scan raw data using a raw data analysis model. The raw data analysis model is a trained machine learning model. The raw data analysis model refers to a model that determines lesion information based on scan raw data. Specifically, the processing device 120 can input the scan raw data to the raw data analysis model, and the raw data analysis model can output the first lesion information. In some embodiments, the scan raw data can be pre-processed to obtain pre-processed scan raw data, and the pre-processed scan raw data can be input to the raw data analysis model for processing. Exemplary pre-processing operations include noise filtering operations, artifact correction and compensation, standardization, normalization, etc.

[0052] In some embodiments, the raw data analysis model can include a machine learning model based on Convolutional Neural Networks (CNN) and / or Transformer model. For example, the raw data analysis model can include a Vision Transformer (ViT) model, a Swin Transformer model, a deep convolutional network model, etc. In some embodiments, the raw data analysis model includes a feature extraction module, a lesion detection module, and a confidence map prediction module. The lesion detection module and the confidence map prediction module are connected with the feature extraction module respectively to receive the feature information output by the feature extraction module in parallel. The feature extraction module is configured to perform feature extraction on the input scan raw data. In some embodiments, the feature extraction module includes a CNN unit, a Transformer encoder. The CNN unit is used to extract multi-scale local feature maps from the scan raw data. The Transformer encoder utilizes the self-attention mechanism to model the global dependency between the local features output by the CNN. The lesion detection module detects the lesion of the target object based on the feature information output by the feature extraction module and outputs a first lesion detection result. The confidence map prediction module is configured to predict the reliability (i.e., confidence) of each raw data point based on the feature information output by the feature extraction module and outputs a first confidence map. In some embodiments, the lesion detection module includes an anchor-based object detection module or an anchor-free object detection module. The confidence map prediction module includes a fully convolutional decoder configured to upsample the feature information to the same spatial size as the scan raw data.

[0053] In some embodiments, the processing device 120 can obtain the raw data analysis model from one or more components of the application scenario 100 of the report generation system (e.g., the storage device 140, the terminal device 130) or an external source through a network (e.g., the network 150). For example, the raw data analysis model is trained in advance by a computing device (e.g., the processing device 120) and stored in a storage device (e.g., the storage device 140). The processing device 120 can access the storage device to obtain the raw data analysis model.

[0054] By way of example only, the processing device 120 can obtain a plurality of first training samples. Each of the first training samples can include sample raw data of a sample subject and first label lesion information. The first label lesion information serves as a label or a gold standard for model training. Similar to the first lesion information, the first label lesion information includes first label lesion detection results. The first label lesion detection results can include whether a lesion exists, a number of lesions, a size of a lesion, a severity of a lesion, a type of a lesion (e.g., solid tumor, cyst, hemorrhage, etc.). In some embodiments, the first label lesion detection results further include a lesion location. In some embodiments, the first label lesion information further includes a first label confidence map. In some embodiments, the first label lesion information is obtained based on a reconstructed image of the sample raw data. In some embodiments, the processing device 120 can perform feature extraction on the reconstructed image of the sample raw data to obtain the first label lesion information. When the first label lesion detection results include the lesion location, the processing device 120 can project the lesion location in the image domain into the raw data domain (e.g., projection domain) by coordinate transformation, generate the lesion location in the raw data domain, and designate the lesion location as the lesion location in the first label lesion detection results. In some embodiments, at least a portion of the first label lesion information can be manually confirmed by a user. For example, the number of lesions, the type of lesions, etc. in the first label lesion information can be manually confirmed by a user.

[0055] Further, the processing device 120 can generate the raw data analysis model by training an initial first initial model based on the plurality of first training samples. The first initial model refers to a machine learning model that is not trained. The first initial model is obtained in a manner similar to the raw data analysis model, and details are not repeated here. In the training process, the sample raw data in the training samples can be used as model input, the first label lesion information corresponding to each of the training samples can be used as training label, and the model parameters of the first initial model can be iteratively updated until an iteration termination condition is satisfied. Any suitable loss function (e.g., MSE) and suitable optimizer (e.g., Adam optimizer) can be used to train the first initial model to obtain the raw data analysis model. In some embodiments, the raw data analysis model is obtained by sequentially or jointly training the first initial model and the second initial model, and details are described in the second initial model section of the present specification Figure 4 , and details are not repeated here.

[0056] The raw data analysis model can learn and identify weak and early signals related to specific lesions (e.g., tumors, hemorrhages). These information are often treated as noise or smoothed and weakened by filtering algorithms in standard image reconstruction processes, so that early abnormalities that are difficult to find by traditional methods can be captured to gain valuable time for clinical intervention.

[0057] At step 330, second lesion information of the target object is determined based on the reconstructed image and the first lesion information. In some embodiments, step 330 can be performed by the processing device 120 or the determining module 220.

[0058] Similar to the first lesion information, the second lesion information refers to information related to the lesion of the target object, which is determined based on the reconstructed image. The second lesion information includes second lesion detection results. The second lesion detection results include whether the lesion is contained, the number of lesions, the size of the lesion, the severity of the lesion, the type of the lesion (e.g., solid tumor, cyst, hemorrhage, etc.), the position of the lesion, etc. The position of the lesion in the second lesion detection results refers to the position of the lesion in the image domain. In some embodiments, the second lesion detection results further include anatomical information of each organ or tissue, for example, the position, size, density, etc. of each organ or tissue.

[0059] In some embodiments, the second lesion information further includes a second confidence map. The second confidence map contains the confidence of each pixel point in the reconstructed image. The confidence of each pixel point is 0-1. The higher the confidence score of a pixel point, the higher the confidence of the pixel point, i.e., the higher the possibility that the signal of the pixel point comes from the anatomical structure of the target object, and the lower the possibility that the signal comes from noise or artifact.

[0060] In some embodiments, the second lesion information is determined by processing the reconstructed image and the first lesion information using an image analysis model. The image analysis model is a trained machine learning model. The image analysis model refers to a model for determining lesion information based on the reconstructed image and the first lesion information. Specifically, the processing device 120 can input the reconstructed image and the first lesion information to the image analysis model, and the image analysis model can output the second lesion information. In some embodiments, the processing device 120 can only input the reconstructed image to the image analysis model, and the image analysis model can output the second lesion information. In some embodiments, other ways can be used to obtain the second lesion information. For example, the reconstructed image can be segmented using an image segmentation algorithm to obtain information such as the position of the lesion, the type of the lesion, the size of the lesion, etc. The type of the lesion, the probability of benign and malignant evaluation, etc. can be determined by manual determination.

[0061] In some embodiments, the image analysis model can include a vision-language large model. For example, the image analysis model includes a ViT or ResNet based vision encoder and a large language model (LLM) text decoder. The vision encoder is configured to analyze anatomical structures, pathological features, and semantic information in the reconstructed image to extract visual features. The LLM text decoder is configured to autoregressively generate a textual description from the visual features. The vision encoder can include a Vision Transformer based model, a convolutional neural network based model, etc. The text decoder can include an autoregressive language model, an encoder-decoder language model, etc.

[0062] In some embodiments, the processing device 120 obtains the image analysis model in a similar manner as obtaining the raw data analysis model. For example only, the processing device 120 can obtain a plurality of second training samples. Each second training sample can include a sample reconstructed image (may also be referred to as a first sample reconstructed image) of a sample subject, sample first lesion information, and second labeled lesion information. The second labeled lesion information is used as a label or a gold standard for model training. The second labeled lesion information includes second labeled lesion detection results. The second labeled lesion detection results include whether a lesion is included, a number of lesions, a size of a lesion, a severity of a lesion, a type of a lesion (e.g., solid tumor, cyst, hemorrhage, etc.), a location of a lesion, etc. In some embodiments, the second labeled lesion detection results further include anatomical structure information of each organ or tissue, e.g., a location, a size, a density, etc. of each organ or tissue. In some embodiments, the second labeled lesion information further includes a second labeled confidence map. In some embodiments, the sample reconstructed image and the sample first lesion information are generated based on scan raw data of the sample subject. For example, the scan raw data of the sample subject can be reconstructed to generate the sample reconstructed image. The scan raw data of the sample subject is input into the raw data analysis model to obtain the sample first lesion information. In some embodiments, the first labeled lesion information is obtained based on a reconstructed image of the sample raw data. In some embodiments, the processing device 120 can perform feature extraction on the sample reconstructed image to obtain the second labeled lesion information. In some embodiments, at least a portion of the second labeled lesion information can be manually confirmed by a user. For example, a number of lesions, a type of a lesion, etc. in the second labeled lesion information.

[0063] Further, the processing device 120 can generate the image analysis model by training an initial second initial model based on the plurality of second training samples. The second initial model refers to a machine learning model that has not been trained. The second initial model can be generated in a similar manner and have a similar structure as the image analysis model, and thus details are not repeated here. During the training process, the sample reconstructed image and the sample first lesion information in the training samples can be used as model input, the second label lesion information corresponding to each training sample can be used as a training label, and the model parameters of the second initial model can be iteratively updated until an iteration termination condition is met. Any suitable loss function (e.g., MSE) and optimizer (e.g., Adam optimizer) can be used to train the second initial model to obtain the image analysis model. In some embodiments, the image analysis model is obtained by sequentially or jointly training the first initial model and the second initial model. For details, please refer to the description of the first initial model. Figure 4 Details are not repeated here.

[0064] At step 340, the processing device 120 or the determination module 220 determines whether the target object contains a lesion region based on the first lesion information and the second lesion information.

[0065] In some embodiments, the processing device 120 determines that the target object contains a lesion in response to at least one of the first lesion information and the second lesion information indicating that there is a lesion and / or the number of lesions is greater than or equal to 1. For example, the processing device 120 determines that the target object contains a lesion in response to the number of lesions in the first lesion detection result being greater than or equal to 1. For another example, the processing device 120 determines that the target object contains a lesion in response to the number of lesions in the second lesion detection result being greater than or equal to 1. For yet another example, the processing device 120 determines that the target object contains a lesion in response to the number of lesions in both the first lesion detection result and the second lesion detection result being greater than or equal to 1. In response to the first lesion information and the second lesion information both indicating that the number of lesions is 0, the processing device 120 determines that the target object does not contain a lesion.

[0066] According to some embodiments of the present application, the first lesion information and the second lesion information are used to determine whether the target object contains a lesion, which effectively reduces the risk of missed diagnosis and misdiagnosis caused by insufficient single image information.

[0067] In response to determining that the target object contains the lesion region, the processing device 120 can perform steps 350-370. In response to determining that the target object does not contain the lesion region, the processing device 120 can generate the structured diagnostic report for the target object based on the first lesion information, the second lesion information, and the clinical information of the target object, using a report generation model. The processing device 120 can generate the structured diagnostic report for the target object in a similar manner as step 370. For example, the first lesion information, the second lesion information, and the clinical information of the target object can be input to the report generation model, and the report generation model can output the structured diagnostic report for the target object. More details about generating the structured diagnostic report can be found in step 370, which will not be repeated here.

[0068] At step 350, in response to determining that the target object contains the lesion region, a target reconstructed image of the lesion region is determined based on the scan raw data and the reconstructed image, using a simulated scan model. In some embodiments, step 350 can be performed by the processing device 120 or the simulated scan module 230.

[0069] The simulated scan model is a trained machine learning model for simulating medical imaging. The simulated scan model can simulate various medical imaging, such as high-resolution imaging, multi-phase scanning, functional imaging, etc. Exemplary high-resolution imaging includes high-resolution computed tomography (HR-CT), high-resolution magnetic resonance imaging (HR-MRI), etc. Exemplary multi-phase scanning includes CT multi-phase scanning, MRI multi-phase dynamic contrast scanning, etc. Exemplary functional imaging includes diffusion-weighted imaging (DWI), perfusion-weighted imaging (PWI), PET scanning, etc. Specifically, the processing device 120 can input the scan raw data and the reconstructed image to the simulated scan model, and the simulated scan model can output the target reconstructed image of the lesion region. In some embodiments, the input to the simulated scan model further includes the first lesion information and the second lesion information.

[0070] In some embodiments, the simulation scan model comprises a plurality of models for simulating different medical imaging. For example, the simulation scan model comprises a first simulation scan model for simulating high resolution imaging, a second simulation scan model for simulating multi-phase scanning, and a third simulation scan model for simulating functional imaging. The processing device 120 can select a suitable simulation scan model according to the first lesion information and the second lesion information. For example, if the lesion type in the first lesion information or the second lesion information is a tumor, the processing device 120 can select the first simulation scan model to obtain a high resolution medical image of the lesion region. For another example, if the lesion type in the first lesion information or the second lesion information is suspected to be a vascular disease (e.g., coronary plaque), the processing device 120 can select the second simulation scan model to obtain a multi-phase enhanced image of the lesion region. For yet another example, if the lesion location in the first lesion information or the second lesion information is located in the myocardium, the processing device 120 can select the third simulation scan model to obtain an MRI myocardial perfusion image and calculate a quantitative perfusion parameter.

[0071] In some embodiments, the above-mentioned plurality of models for simulating different medical imaging (e.g., the first simulation scan model, the second simulation scan model, and the third simulation scan model) can be integrated into a comprehensive simulation scan model. The comprehensive simulation scan model comprises a model selector. The model selector is connected to the above-mentioned plurality of models respectively. The first lesion information, the second lesion information, the scan raw data, and the reconstructed image can be input into the model selector. The model selector determines the model to be used based on the first lesion information and the second lesion information, and inputs the scan raw data and the reconstructed image into the selected model to generate the target reconstructed image. For example, the model selector is connected to the first simulation scan model, the second simulation scan model, and the third simulation scan model respectively. The first lesion information, the second lesion information, the scan raw data, and the reconstructed image are input into the model selector. The model selector determines that the first simulation scan model is to be used based on the first lesion information and the second lesion information. Then, the model selector inputs the scan raw data and the reconstructed image into the first simulation scan model to generate a high resolution medical image.

[0072] In some embodiments, the simulation scan model can comprise various deep learning models. For example, the simulation scan model can comprise a generative adversarial network (GAN), a diffusion model, a conditional variational autoencoder (CVAE), nnUNet, Mask R-CNN, a time series analysis model (e.g., 3DCNN, LSTM), etc.

[0073] In some embodiments, the processing device 120 obtains the simulated scan model in a similar way as obtaining the live data analysis model. For example only, the processing device 120 can obtain a plurality of third training samples. Each of the third training samples can include sample live data of a sample subject, a sample reconstructed image, and a reference reconstructed image. The reference reconstructed image serves as a label or a gold standard for model training. In some embodiments, the sample reconstructed image is generated based on the sample live data. The reference reconstructed image can be generated by performing medical imaging on the sample subject using the medical imaging device 110. For example, for a first simulated scan model, the reference reconstructed image can be generated by performing high resolution imaging on the sample subject using the medical imaging device 110 and based on the obtained scan live data reconstruction. For another example, for a second simulated scan model, the reference reconstructed image can be generated by performing multi-phase scan on the sample subject using the medical imaging device 110 and based on the obtained scan live data reconstruction. For yet another example, for a third simulated scan model, the reference reconstructed image can be generated by performing functional imaging on the sample subject using the medical imaging device 110 and based on the obtained scan live data reconstruction. Further, the processing device 120 can generate the simulated scan model by training a third initial model based on the plurality of third training samples. The third initial model refers to a machine learning model that is not trained. The third initial model is obtained in a similar way and has a similar structure as the simulated scan model, which will not be repeated here. During the training process, the sample reconstructed image and the sample live data in the training sample can be used as model input, the reference reconstructed image corresponding to each training sample can be used as a training label, and the model parameters of the third initial model can be iteratively updated until an iteration termination condition is satisfied. Any suitable loss function (e.g., MSE) and suitable optimizer (e.g., Adam optimizer) can be used to train the third initial model to obtain the simulated scan model.

[0074] In a conventional procedure, after obtaining the reconstructed image, it is necessary to pass through the judgment of the radiology doctor and the clinical doctor in sequence to determine whether additional scanning (e.g., high resolution scanning, enhanced scanning, etc.) for the lesion is needed. If it is determined that additional scanning for the lesion is needed, the examination order is issued by the clinical doctor, and then the additional scanning is performed. This is a serial and artificial-dependent procedure, resulting in a long diagnosis and treatment cycle and low efficiency. After determining the lesion region, the present specification directly triggers simulated scanning through the simulated scan model without actual scanning, greatly improving the diagnosis efficiency. By replacing the additional physical scanning with simulated scanning, the target subject is free from unnecessary radiation exposure, contrast agent injection risk, and additional economic burden.

[0075] At step 360, feature extraction is performed on the target reconstructed image to obtain third lesion information. In some embodiments, step 360 can be performed by the processing device 120 or the determination module 220.

[0076] The third lesion information refers to information related to the lesion of the target object. The third lesion information includes a third lesion detection result. The third lesion detection result includes a lesion size, a lesion severity, a lesion type (e.g., a solid tumor, a cyst, a hemorrhage, etc.), a lesion position, a lesion density, and the like.

[0077] In some embodiments, the third lesion information further includes a third confidence map. The third confidence map contains a confidence of each pixel point in the target reconstructed image. The confidence of each pixel point is 0-1. The higher the confidence score of a pixel point, the higher the confidence of the pixel point, i.e., the higher the possibility that the signal of the pixel point comes from the anatomical structure of the target object and the lower the possibility that the signal comes from noise or artifacts.

[0078] In some embodiments, the third lesion information is determined by processing the target reconstructed image using a lesion analysis model. The lesion analysis model is a trained machine learning model. The lesion analysis model refers to a model for determining lesion information based on a target reconstructed image. Specifically, the processing device 120 can input the target reconstructed image to the lesion analysis model, and the lesion analysis model can output the third lesion information. In some embodiments, other ways can be used to obtain the third lesion information. For example, an image segmentation algorithm can be used to segment the target reconstructed image to obtain information such as lesion position, lesion type, lesion size, and the like. The lesion type, lesion benignity and malignancy evaluation probability, and the like can be determined by human.

[0079] In some embodiments, the lesion analysis model can be similar to the image analysis model. The lesion analysis model can adopt the structure of the image analysis model, for example, the lesion analysis model includes a visual-linguistic large model.

[0080] In some embodiments, the processing device 120 obtains the lesion analysis model in a similar way as obtaining the raw data analysis model. For example only, the processing device 120 can obtain a plurality of fourth training samples. Each of the fourth training samples can include a second sample reconstructed image of a sample subject and third labeled lesion information. The third labeled lesion information serves as a label or a gold standard for model training. In some embodiments, the second sample reconstructed image can be generated in a similar way as generating the reference reconstructed image in step 350. The third labeled lesion information can be manually confirmed by a user. Further, the processing device 120 can generate the lesion analysis model by training a fourth initial model based on the plurality of fourth training samples. The fourth initial model refers to a machine learning model that is not trained. The fourth initial model can be obtained in a similar way and have a similar structure as the lesion analysis model, which will not be repeated here. During the training process, the second sample reconstructed image in each of the training samples can be used as a model input, the third labeled lesion information corresponding to each of the training samples can be used as a training label, and the model parameters of the fourth initial model can be iteratively updated until an iteration termination condition is satisfied. Any suitable loss function (e.g., MSE) and optimizer (e.g., Adam optimizer) can be used to train the fourth initial model to obtain the lesion analysis model.

[0081] At step 370, a structured diagnostic report of the target subject is generated based on the first lesion information, the second lesion information, the third lesion information, and the clinical information, using a report generation model. In some embodiments, step 370 can be performed by the processing device 120 or the report generation module 240.

[0082] The structured diagnostic report of the target subject includes a diagnostic result. The diagnostic result includes at least a lesion assessment result. The lesion assessment result refers to a final judgment on the nature of the lesion, for example, the lesion assessment result can be that the target subject has primary lung adenocarcinoma, liver cyst, benign nodule, etc. In some embodiments, the diagnostic result further includes one or more of a diagnostic confidence index, a clinical action recommendation, etc. The diagnostic confidence index refers to a quantitative confidence score (e.g., 95%) indicating the degree of confidence of the report generation model in the diagnostic result. The clinical action recommendation refers to a proposed follow-up treatment recommendation for the lesion, for example, recommending surgical resection, recommending regular follow-up, recommending further performing PET-CT examination, etc.

[0083] In some embodiments, the structured diagnosis further comprises a provenance, the provenance indicating information relied on to derive the diagnosis. For example, the provenance of the lesion size is the average of the lesion size in the first lesion information, the second lesion information, and the third lesion information. For another example, the lesion type is the lesion type in the third lesion information. For yet another example, the provenance of the lesion size is the weighted average of the lesion severity in the first lesion information, the second lesion information, and the third lesion information. In some embodiments, the structured diagnosis further comprises a consistency detection result. The consistency detection result can indicate whether the first lesion detection result, the second lesion detection result, and the third lesion detection result are in conflict, i.e., whether the first lesion detection result, the second lesion detection result, and the third lesion detection result are consistent.

[0084] The report generation model is a trained machine learning model. The report generation model refers to a model used to generate a structured diagnosis report. Specifically, the processing device 120 can input the first lesion detection result, the second lesion detection result, the third lesion detection result, and the clinical data to the report generation model, and the report generation model can output the structured diagnosis report.

[0085] In some embodiments, the processing device 120 determines a first weight corresponding to the first lesion detection result, a second weight corresponding to the second lesion detection result, and a third weight corresponding to the third lesion detection result based on the clinical information, the first confidence map, the second confidence map, and the third confidence map. Each of the first weight, the second weight, and the third weight has a value between 0 and 1. The sum of the first weight, the second weight, and the third weight can not equal 1.

[0086] Specifically, Figure 5 is a schematic diagram of an example process of determining the first weight, the second weight, and the third weight according to some embodiments of the present specification. As shown in FIG. 8, the processing device 120 can determine the first weight, the second weight, and the third weight based on the clinical information, the first confidence map, the second confidence map, and the third confidence map. Figure 5As shown, the processing device 120 can determine the case type of the target object based on the clinical information. The case type can be classified in different manners. For example, the case type can include lung cancer, liver cancer, vascular disease (e.g., embolism, stenosis), trauma, hemorrhage, etc. according to lesion type and location. For another example, the case type can include early lesion, medium lesion, and late lesion according to disease course. For example, the doctor's preliminary diagnosis result in the clinical information contains the information of "suspected lung cancer", and the processing device 120 can determine the case type of the target object as lung cancer. For another example, the current symptom information in the clinical information contains the information of "sudden chest pain, D-dimer elevated", and the processing device 120 can determine the case type of the target object as pulmonary embolism. For another example, the current symptom information and past medical history in the clinical information contain the information of "history of hepatitis B, AFP elevated", and the processing device 120 can determine the case type of the target object as liver cancer. For another example, the case type can include common disease and rare disease according to common degree.

[0087] Further, the processing device 120 can determine the preset first weight corresponding to the first lesion detection result, the preset second weight corresponding to the second lesion detection result, and the preset third weight corresponding to the third lesion detection result based on the case type. For example, if the case type of the target object is lung cancer, since the morphological analysis of the reconstructed image by the image analysis model is more accurate for the diagnosis of lung cancer, the preset second weight can be assigned a higher value, e.g., 1. For another example, if the case type of the target object is liver cancer, since the perfusion information or enhanced kinetics information provided by functional imaging or enhanced scanning is more accurate for the diagnosis of liver cancer, the preset third weight can be assigned a higher value, e.g., 1. For another example, if the case type of the target object is early lesion, the early signal detection capability provided by the raw data model can be more accurate for the diagnosis of early lesion, and the preset first weight can be assigned a higher value, e.g., 1. In some embodiments, the mapping relationship between different disease types and weight assignment schemes (i.e., the preset weights corresponding to the first, second, and third lesion detection results respectively) is preset. The processing device can determine the preset first weight, the preset second weight, and the preset third weight based on the mapping relationship.

[0088] Then, the processing device 120 can adjust the preset first weight, the preset second weight, and the preset third weight based on the first confidence map, the second confidence map, and the third confidence map to determine the first weight, the second weight, and the third weight. In some embodiments, the processing device 120 determines a first average confidence based on the first confidence map. The first average confidence can be an average of the confidence of all the data points. If the first average confidence is lower than a first confidence threshold, the processing device 120 adjusts the preset first weight lower; if the first average confidence is higher than a second confidence threshold, the processing device 120 adjusts the preset first weight higher; if the first average confidence is between the first confidence threshold and the second confidence threshold, the processing device 120 specifies the preset first weight as the first weight. The processing device 120 determines a second average confidence based on the second confidence map. The second average confidence can be an average of the confidence of all the pixels in the reconstructed image. Alternatively, the second average confidence can be an average of the confidence of the pixels in the reconstructed image corresponding to the lesion region. If the second average confidence is lower than a third confidence threshold, the processing device 120 adjusts the preset second weight lower; if the second average confidence is higher than a fourth confidence threshold, the processing device 120 adjusts the preset second weight higher; if the second average confidence is between the third confidence threshold and the fourth confidence threshold, the processing device 120 specifies the preset second weight as the second weight. The processing device 120 determines a third average confidence based on the third confidence map. The third average confidence can be an average of the confidence of the pixels in the target reconstructed image corresponding to the lesion region. If the third average confidence is lower than a fifth confidence threshold, the processing device 120 adjusts the preset third weight lower; if the third average confidence is higher than a sixth confidence threshold, the processing device 120 adjusts the preset third weight higher; if the third average confidence is between the fifth confidence threshold and the sixth confidence threshold, the processing device 120 specifies the preset third weight as the third weight. In some embodiments, the processing device 120 can obtain a fourth weight corresponding to the clinical information. The fourth weight can be set as needed. In some embodiments, since the clinical information is confirmed information, the fourth weight can be 1.

[0089] Further, the processing device 120 can generate a structured diagnostic report of the target object by processing the first lesion detection result, the second lesion detection result, the third lesion detection result, the first weight, the second weight, the third weight, the fourth weight, and the clinical information using a report generation model. Specifically, the processing device 120 can input the first lesion detection result, the second lesion detection result, the third lesion detection result, the first weight, the second weight, the third weight, the fourth weight, and the clinical information to the report generation model, and the report generation model can output the structured diagnostic report.

[0090] In some embodiments, in response to the existence of the historical diagnosis data related to the lesion region, the processing device 120 can determine the lesion change information based on the historical diagnosis data, the first lesion information, the second lesion information, and the third lesion information. Exemplary lesion change information includes a size change of the lesion, a number change of the lesion, a type change of the lesion, a location change of the lesion, a severity change of the lesion, and the like. Further, the processing device 120 processes the first lesion information, the second lesion information, the third lesion information, and the lesion change information using the report generation model to generate the structured diagnosis report of the target object. For example, the processing device 120 can input the first lesion detection result, the second lesion detection result, the third lesion detection result, the first weight, the second weight, the third weight, the fourth weight, and the clinical information and the lesion change information to the report generation model, and the report generation model can output the structured diagnosis report. In combination with the historical diagnosis data of the lesion region, by adding the lesion change information as the input of the report generation model, a more accurate diagnosis report can be obtained.

[0091] In some embodiments, the report generation model includes a feature fusion module and a structured report generation module. The feature fusion module is configured to determine the consistency detection result, the fused feature information, and the initial traceability basis corresponding thereto by performing consistency detection and feature fusion based on the first lesion detection result, the second lesion detection result, the third lesion detection result, the first weight, the second weight, the third weight, the fourth weight, and the clinical information. The structured report generation module is configured to determine the diagnosis result based on the fused feature information, and determine the traceability basis based on the initial traceability basis. The structured report generation module is further configured to generate the structured diagnosis report based on the consistency detection result, the diagnosis result, and the traceability basis.

[0092] Figure 6 is a schematic diagram of an exemplary process of determining a structured diagnosis report using a report generation model according to some embodiments of the present specification. As shown in Figure 6As shown, the processing device 120 can input the first lesion detection result, the second lesion detection result, the third lesion detection result, the first weight, the second weight, and the third weight into a feature fusion module. The feature fusion module can perform consistency detection on each feature item in the first lesion detection result, the second lesion detection result, and the third lesion detection result to generate a consistency detection result. The feature items include the number of lesions, the size of lesions, the severity of lesions, the type of lesions, the position of lesions, etc. The consistency detection result can indicate whether the results of the corresponding same feature items in the first lesion detection result, the second lesion detection result, and the third lesion detection result are consistent (or exist conflicts). For example, the size of lesions and the position of lesions in the second lesion detection result and the third lesion detection result are not the same, and thus the consistency detection result contains that the size of lesions and the position of lesions are inconsistent (i.e., conflicts). For another example, for the same lesion, the type of lesions in the first lesion detection result, the second lesion detection result, and the third lesion detection result is the same, and thus the consistency detection result contains that the type of lesions is consistent (i.e., no conflicts). For yet another example, for the same lesion, the difference between the severity of lesions in the first lesion detection result, the second lesion detection result, and the third lesion detection result is less than a threshold, and thus the consistency detection result contains that the severity of lesions is consistent (i.e., no conflicts). In some embodiments, the feature fusion module can align the corresponding feature items in the first lesion detection result, the second lesion detection result, and the third lesion detection result. Further, for each feature item, the feature fusion module can perform weighted fusion according to the value of the feature item in the first lesion detection result, the second lesion detection result, the third lesion detection result, and the clinical information and the corresponding first weight, second weight, third weight, and fourth weight to obtain the fusion information of the feature item. Then, the feature fusion module splices the fusion information of each feature item to generate the fusion feature information. The initial traceability basis includes the source of the fusion information corresponding to each feature item and the weight at the time of fusion.

[0093] Then, the consistency detection result, the fused feature information, and the initial provenance basis can be input into a structured report generation module, which can output a structured report. Specifically, the structured report generation module can determine a diagnosis result based on the fused feature information. For example, the structured report generation module can specify the lesion size, the lesion location, and the lesion type in the fused feature information as the lesion size, the lesion location, and the lesion type in the diagnosis result. For another example, the structured report generation module can determine whether the lesion is a benign lesion or a malignant lesion according to the severity of the lesion. For another example, the structured report generation module can determine whether the lesion belongs to a benign lesion or a malignant lesion based on one or more of the lesion size, the severity of the lesion, and the lesion type. For example, the greater the severity of the lesion and the greater the size, the lesion can be determined to be a malignant lesion. For another example, for a cyst, it can be determined to be a benign lesion. In some embodiments, the structured report generation module can determine the diagnosis result by referring to a domain knowledge base, medical guidelines, medical standards, etc.

[0094] The structured report generation module can determine a provenance basis based on the initial provenance basis. Specifically, the provenance basis includes the sources of the fused information whose weights exceed a certain threshold in the initial provenance basis. For example, the initial provenance basis indicates that the lesion type comes from the first lesion detection result, the second lesion detection result, and the third lesion detection result, and the weights are all 1, then the provenance basis indicates that the lesion type comes from the first lesion detection result, the second lesion detection result, and the third lesion detection result. For another example, the initial provenance basis indicates that the lesion location comes from the second lesion detection result and the third lesion detection result, and the weights are all 1, then the provenance basis indicates that the lesion location comes from the second lesion detection result and the third lesion detection result. For another example, the initial provenance basis indicates that the lesion size comes from the second lesion detection result and the third lesion detection result, and the weights are 0.4 and 0.8 respectively, then the provenance basis indicates that the lesion size comes from the third lesion detection result. For example, the initial provenance basis indicates that the lesion severity comes from the first lesion detection result, the second lesion detection result, and the third lesion detection result, and the weights are 0.2, 0.4, and 0.9 respectively, then the provenance basis indicates that the lesion severity comes from the third lesion detection result.

[0095] Finally, the structured report generation module arranges the consistency detection result, the diagnosis result, and the provenance basis according to a preset format to generate a structured diagnosis report.

[0096] In some embodiments, the feature fusion module can include a multi-layer perception, a deep neural network, a gradient boosting decision tree model, a Bayesian network, etc. The structured report generation module can include a Transformer-based autoregressive language model, a retrieval-augmented generation model, a large language model, etc.

[0097] According to some embodiments of the present application, the consistency detection result, the diagnosis result and the traceability basis are jointly integrated into the structured diagnosis report, greatly improving the credibility, traceability and practicality of the structured diagnosis report. The decision-making process of the model is transparentized through the traceability basis, eliminating the "black box" doubts. The uncertainty is actively revealed through the consistency detection, alerting the risk and inspiring human-machine collaborative review. Finally, the clear diagnosis result output enables doctors to quickly focus on key information, greatly improving the reliability of diagnosis efficiency, and providing a complete audit trail for the compliance supervision and continuous optimization of the model.

[0098] In some embodiments, the processing device 120 acquires the report generation model in a manner similar to acquiring the living data analysis model. Merely by way of example, the processing device 120 can acquire a plurality of fifth training samples. Each fifth training sample can include a sample first lesion detection result and its corresponding sample first weight, a sample second lesion detection result and its corresponding sample second weight, a sample third lesion detection result and its corresponding sample third weight, sample clinical information of the sample object and its corresponding sample fourth weight, and a reference structured diagnosis report. The reference structured diagnosis report serves as a label or gold standard for model training. In some embodiments, the sample first lesion detection result can be generated in a manner similar to generating the first lesion information in step 320. In some embodiments, the sample second lesion detection result can be generated in a manner similar to generating the second lesion information in step 330. In some embodiments, the sample third lesion detection result can be generated in a manner similar to generating the third lesion information in step 360. In some embodiments, the sample first weight, the sample second weight, the sample third weight and the sample fourth weight can be determined in a manner similar to generating the first weight, the second weight, the third weight and the fourth weight. The reference structured diagnosis report can be manually confirmed by a user.

[0099] Further, the processing device 120 can generate the report generation model by training an initial fifth initial model based on the plurality of fifth training samples. The fifth initial model refers to a machine learning model that has not been trained. The fifth initial model is acquired in a manner and structure similar to the report generation model, which will not be described here. In the training process, the sample first lesion detection result, the sample first weight, the sample second lesion detection result, the sample second weight, the sample third lesion detection result, the sample third weight, the sample clinical information and the sample fourth weight in the training sample can be used as model input, the reference structured diagnosis report corresponding to each training sample can be used as a training label, and the model parameters of the fifth initial model can be iteratively updated until the iteration termination condition is met. Any suitable loss function (e.g., MSE) and suitable optimizer (e.g., Adam optimizer) can be used to train the fifth initial model to obtain the report generation model.

[0100] It should be noted that the above description of process 300 is for illustrative purposes only and does not limit the scope of this specification. Those skilled in the art can make various modifications and changes to process 300 under the guidance of this specification. However, these modifications and changes remain within the scope of this specification.

[0101] Figure 4 This is a schematic diagram illustrating the training process of exemplary raw data analysis models and image analysis models according to some embodiments of this specification. Figure 4 It can be executed by the processing device 120 or the model training module 250.

[0102] Live data analysis models and image analysis models can be obtained by sequentially training a first initial model and a second initial model. For example... Figure 4 As shown, the processing device 120 can acquire raw sample data of a sample object, a reconstructed image of the sample corresponding to the raw sample data, and first-label lesion information and second-label lesion information corresponding to the reconstructed image. Further, the processing device 120 trains a first initial model using the raw sample data and the first-label lesion information to obtain a raw data analysis model. Then, the processing device 120 can determine the first lesion information of the sample corresponding to the raw sample data based on the raw sample data and the raw data analysis model. Specifically, the raw sample data can be input into the trained raw data analysis model to obtain the first lesion information. Finally, the processing device 120 trains a second initial model using the reconstructed image, the first lesion information, and the second-label lesion information to obtain an image analysis model. For a more detailed description of training the first initial model to generate the raw data model, please refer to step 320. For a more detailed description of training the second initial model to generate the image analysis model, please refer to step 330. In some embodiments, the processing device 120 can acquire raw sample data of a sample object and generate a reconstructed image of the sample based on the raw sample data. Further, the processing device 120 performs feature extraction on the reconstructed sample image to obtain first-label lesion information and second-label lesion information. In some embodiments, at least a portion of the first-label lesion information and the second-label lesion information can be determined manually. For a more detailed description of the first-label lesion information and the second-label lesion information, please refer to steps 320 and 330.

[0103] In some embodiments, the raw data analysis model and the image analysis model can be obtained by jointly training the first initial model and the second initial model. The training of the first initial model and the second initial model includes a plurality of iteration processes. In each iteration process, the processing device 120 inputs the sample raw data into the first initial model, and the first initial model outputs the predicted first lesion information. Subsequently, the processing device 120 compares the predicted first lesion information with the first labeled lesion information, and determines the first loss value. Further, the processing device 120 inputs the sample reconstructed image and the predicted first lesion information into the second initial model, and the second initial model outputs the predicted second lesion information. Subsequently, the processing device 120 compares the predicted second lesion information with the second labeled lesion information, and determines the second loss value. The processing device 120 can update the first initial model and the second initial model based on the first loss value and the second loss value.

[0104] After a plurality of iterations, until a termination condition of the model training is satisfied, the processing device 120 can stop the model training, and specify the first initial model and the second initial model obtained in the last iteration as the raw data analysis model and the image analysis model. An exemplary termination condition can include that the value of the loss function is less than a threshold, the number of iterations meets a requirement, etc.

[0105] Figure 7 is a schematic diagram of an exemplary process of generating a structured diagnostic report according to some embodiments of the present specification.

[0106] As shown in Figure 7 , the process of generating a structured diagnostic report includes: obtaining scan raw data of a target object; generating a reconstructed image of the target object based on the scan raw data; inputting the scan raw data into a raw data analysis model, and the raw data analysis model outputs first lesion information; inputting the reconstructed image and the first lesion information into an image analysis model, and the image analysis model outputs second lesion information; determining whether the target object contains a lesion region based on the first lesion information and the second lesion information; in response to determining that the target object contains a lesion region, inputting the scan raw data and the reconstructed image into a simulated scan model, and the simulated scan model outputs a target reconstructed image; inputting the target reconstructed image into a lesion analysis model, and the lesion analysis model outputs third lesion information; inputting the first lesion information, the second lesion information, the third lesion information, and clinical information into a report generation model, and the report generation model outputs a structured diagnostic report. In some embodiments, in response to the target object having historical diagnostic data related to the lesion region, further determining lesion change information based on the historical diagnostic data, the first lesion information, the second lesion information, and the third lesion information; inputting the first lesion information, the second lesion information, the third lesion information, the clinical information, and the lesion change information into the report generation model, and the report generation model outputs a structured diagnostic report.

[0107] In some embodiments of the present specification, the first lesion information can be obtained using a raw data analysis model, the second lesion information can be obtained using an image analysis model, the third lesion information can be obtained using a lesion analysis model, and finally, a structured diagnostic report can be generated using a report generation model based on the first lesion information, the second lesion information, the third lesion information, and the clinical information. The beneficial effects brought by the embodiments of the present specification can include but are not limited to: (1) The raw data analysis model can learn and identify weak and early signals related to specific lesions (such as tumors, hemorrhages). These information are often considered as noise or smoothed and weakened by filtering algorithms in the standard image reconstruction process, so that early abnormalities that are difficult to find by traditional methods can be captured, and valuable time for clinical intervention can be saved. (2) In the conventional process, after obtaining the reconstructed image, the image doctor and the clinician need to judge in sequence to determine whether additional scanning (for example, high-resolution scanning, enhanced scanning, etc.) is needed for the lesion. If it is determined that additional scanning for the lesion is needed, the clinician needs to issue an examination order, and then additional scanning is performed. This is a serial and artificial process, which leads to long diagnosis and treatment cycle and low efficiency. After determining the lesion area according to the present specification, the simulated scanning model is triggered to perform simulated scanning without actual scanning, which greatly improves the diagnosis efficiency. By replacing the additional physical scanning with simulated scanning, the target object is free from unnecessary radiation exposure, contrast agent injection risk and additional economic burden. (3) According to some embodiments of the present application, the structured diagnostic report is generated based on the first lesion information, the second lesion information, the third lesion information and the clinical information, which can complement and verify each other, significantly improve the robustness and accuracy of diagnosis, and reduce the risk of missed diagnosis and misdiagnosis caused by single information. (4) According to some embodiments of the present application, the consistency detection result, the diagnosis result and the traceability basis are integrated into the structured diagnostic report, which greatly improves the credibility, traceability and practicability of the structured diagnostic report. The decision-making process of the model is transparent through the traceability basis, eliminating the "black box" doubts. The uncertainty is actively revealed through the consistency detection, which alerts the risk and stimulates the human-machine collaborative review. Finally, the clear diagnosis result output enables the doctor to quickly focus on the key information, greatly improves the reliability of diagnosis efficiency, and provides a complete audit trail for the compliance supervision and continuous optimization of the model.

[0108] The above has described the basic concepts. Obviously, for those skilled in the art, the above detailed disclosure is only an example and does not limit the present specification. Although it is not explicitly stated here, those skilled in the art can make various modifications, improvements and corrections to the present specification. Such modifications, improvements and corrections are suggested in the present specification, so such modifications, improvements and corrections still belong to the spirit and scope of the exemplary embodiments of the present specification.

[0109] Also, the use of "a" or "an" or "the" or "at least one" or "one or more" or "one or more" are used to include one or more than one, independent of other instances or usages of "at least one" or "one or more." The use of the conjunctive, "or" in the context of a list of items prefaced by "for example" or "such as" indicates a non-exclusive list, and the list of items can include one item, more than one item, or the same term can be repeated in the list. The term "adapted to" is used to indicate that the item being adapted to is changed to operate intelligently, and is not a manual operation, unless otherwise indicated by context.

[0110] In addition, the order of presentation of the treatment elements and sequences described in this specification, the use of the numerals, or the use of other designations, is not intended to limit the order of the processes and methods of this specification, unless the order is explicitly stated in the claims. Although some presently preferred embodiments of the application have been described above with particular emphasis on the uses thereof, it will be understood that this is for illustrative purposes only and that additional modifications and equivalents of the embodiments disclosed can be used. For example, although the system components described above can be implemented by hardware devices, they can also be implemented by software solutions, such as installing the described system on existing servers or mobile devices.

[0111] Similarly, it is noted that while the present specification has described certain embodiments with particular emphasis, in order to simplify the description and to assist in the understanding of one or more inventive embodiments, features of the embodiments described in the foregoing description can sometimes be presented in a causal relationship, or in a sequence, or in parallel, or in some other temporal relationship, to other features. However, this disclosure does not require that the described features be related in that manner, or occur in that order, or in parallel, or at that time. Nor is embodiment of the specification required to have each feature of the embodiments described in the foregoing description.

[0112] Some embodiments use numerical designations to describe components, quantities of attributes. It is understood that such numerical designations used in the description of embodiments are, in some examples, modified by the adjectives "about," "approximately," or "generally." Unless otherwise stated, "about," "approximately," or "generally" indicates that the stated numerical value is permitted to vary by ±20%. Accordingly, numerical parameters in the description and claims are approximations, and can vary depending upon the requirements of the particular embodiment. In some embodiments, numerical parameters are determined by the use of standard techniques. Although the numerical ranges and parameters setting forth the broad scope of the embodiments of the specification are approximations, unless otherwise indicated, in specific embodiments, numerical values are reported as precisely as practicable. The numerical values set forth in the specific examples are reported as precisely as practicable.

[0113] Each patent, patent application, patent publication, and other material cited in this specification is hereby incorporated by reference in its entirety herein for the teachings relevant to the sentence and / or paragraph in which the reference is presented. Document histories, to the extent not inconsistent with the pertinent U.S. patent application file history, are also incorporated by reference herein. To the extent that material incorporated by reference contradicts or contradicts any portion of this specification, including definition, the portion of the material incorporated by reference prevails. Note, however, that in the event of inconsistencies between any such material and the present specification, including definitions, the present specification, including definitions, will control.

[0114] Finally, it should be understood that the embodiments described herein are merely exemplary of the principles of the present description. Other embodiments can be devised without departing from the scope of the present description. Accordingly, the embodiments described herein are not intended to limit the scope of the present description, but rather are intended to be exemplary thereof.

Claims

1. A method of generating a diagnostic report, characterized by, The method comprises: obtaining scan raw data of a target object, a reconstructed image of the target object, and clinical information of the target object, the scan raw data being original data obtained by scanning the target object by a medical imaging device, and the reconstructed image being generated based on the scan raw data; determining first lesion information of the target object based on the scan raw data; determining second lesion information of the target object based on the reconstructed image and the first lesion information; determining whether the target object contains a lesion area based on the first lesion information and the second lesion information; in response to determining that the target object contains a lesion area, determining a target reconstructed image of the lesion area based on the scan raw data and the reconstructed image using a simulation scanning model; performing feature extraction on the target reconstructed image to obtain third lesion information; and generating a structured diagnostic report of the target object by using a report generation model based on the first lesion information, the second lesion information, the third lesion information, and the clinical information.

2. The method of claim 1, wherein: the first lesion information is determined by processing the scan raw data using a raw data analysis model; the second lesion information is determined by processing the reconstructed image and the first lesion information using an image analysis model; and the third lesion information is determined by processing the target reconstructed image using a lesion analysis model, wherein the raw data analysis model, the image analysis model, and the lesion analysis model are trained machine learning models. The raw data analysis model and the image analysis model are obtained by:

3. The method of claim 2, wherein, obtaining sample raw data, a sample reconstructed image corresponding to the sample raw data, first labeled lesion information corresponding to the sample reconstructed image, and second labeled lesion information; obtaining the raw data analysis model by training a first initial model using the sample raw data and the first labeled lesion information; determining sample first lesion information corresponding to the sample raw data based on the sample raw data and the raw data analysis model; and obtaining the image analysis model by training a second initial model using the sample reconstructed image, the sample first lesion information, and the second labeled lesion information. In response to the existence of historical diagnostic data related to the lesion area, the method further comprises: determining lesion change information based on the historical diagnostic data, the first lesion information, the second lesion information, and the third lesion information; 4. The method of claim 1, wherein, the generating of the structured diagnostic report of the target object by using the report generation model based on the first lesion information, the second lesion information, the third lesion information, and the clinical information comprises: processing the first lesion information, the second lesion information, the third lesion information, the clinical information, and the lesion change information by using the report generation model to generate the structured diagnostic report of the target object. ​ ​ 5. The method of claim 1, wherein, The first lesion information includes a first lesion detection result and a first confidence map, the second lesion information includes a second lesion detection result and a second confidence map, the third lesion information includes a third lesion detection result and a third confidence map, and The generating of the structured diagnostic report of the target object by using the report generation model based on the first lesion information, the second lesion information, the third lesion information and the clinical information comprises: determining a first weight corresponding to the first lesion detection result, a second weight corresponding to the second lesion detection result and a third weight corresponding to the third lesion detection result based on the clinical information, the first confidence map, the second confidence map and the third confidence map; and processing the first lesion detection result, the second lesion detection result, the third lesion detection result, the first weight, the second weight and the third weight by using the report generation model to generate the structured diagnostic report of the target object.

6. The method of claim 5, wherein, The determining of the first weight corresponding to the first lesion detection result, the second weight corresponding to the second lesion detection result and the third weight corresponding to the third lesion detection result based on the clinical information, the first confidence map, the second confidence map and the third confidence map comprises: determining a case type of the target object based on the clinical information; determining a preset first weight corresponding to the first lesion detection result, a preset second weight corresponding to the second lesion detection result and a preset third weight corresponding to the third lesion detection result based on the case type; adjusting the preset first weight, the preset second weight and the preset third weight based on the first confidence map, the second confidence map and the third confidence map to determine the first weight, the second weight and the third weight.

7. The method of claim 5, wherein, The structured diagnostic report of the target object comprises a diagnostic result and a trace basis, and the trace basis indicates information on which the diagnostic result is based.

8. The method of claim 7, wherein, The report generation model comprises the following modules: a feature fusion module configured to determine a consistency detection result, fused feature information and an initial trace basis corresponding thereto by performing consistency detection and feature fusion based on the first lesion detection result, the second lesion detection result, the third lesion detection result, the first weight, the second weight, the third weight and the clinical information; a structured report generation module configured to: determine the diagnostic result based on the fused feature information; determine the trace basis based on the initial trace basis; generate the structured diagnostic report based on the consistency detection result, the diagnostic result and the trace basis.

9. A system for generating a diagnostic report, the system comprising: comprise: an acquisition module configured to acquire scan raw data of a target object, a reconstructed image of the target object and clinical information of the target object, the scan raw data being original data obtained by scanning the target object by a medical imaging device, and the reconstructed image being generated based on the scan raw data; a determination module configured to: determine first lesion information of the target object based on the raw scan data; determine second lesion information of the target object based on the reconstructed image and the first lesion information; determine whether the target object contains a lesion based on the first lesion information and the second lesion information; a simulation scanning module configured to, in response to determining that the target object contains a lesion region, determine a target reconstructed image of the lesion region based on the raw scan data and the reconstructed image using a simulation scanning model; the determination module is further configured to: perform feature extraction on the target reconstructed image to obtain third lesion information; and a report generation module configured to generate a structured diagnostic report of the target object based on the first lesion information, the second lesion information, the third lesion information, and the clinical information using a report generation model.

10. A system for generating a diagnostic report, the system comprising: comprise: at least one storage device for storing computer instructions; at least one processor for executing the computer instructions to implement the method of any one of claims 1-8.

Images