Image processing method, computer-aided diagnosis method for fatty liver, device, system, computer storage medium, and computer program product

Abstract

The present application provides an image processing method, a computer-aided diagnosis method for fatty liver, a device, a system, a computer storage medium, and a computer program product. The image processing method comprises: acquiring a medical image, and performing feature extraction on the medical image to obtain deep image features; performing encoding processing on the deep image features to obtain image texture features, wherein the image texture features characterize a pixel value distribution of respective pixels in the medical image; fusing the deep image features and the image texture features to obtain visual features corresponding to the medical image; and performing detection result prediction on the basis of the visual features to obtain a predicted detection result. The present application enables acquisition of accurate detection results.

Description

Image processing methods, computer-aided diagnostic methods for fatty liver, equipment, systems, computer storage media, and computer program products

[0001] This disclosure claims priority to Chinese Patent Application No. 202411205413.3, filed with the China Patent Office on August 29, 2024, entitled "Image Processing Method, Computer-Aided Diagnosis Method, Device, System, Computer Storage Medium and Computer Program Product", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This disclosure relates to the field of computer technology, and in particular to an image processing method, a computer-aided diagnostic method for fatty liver, an apparatus, a system, a computer storage medium, and a computer program product. Background Technology

[0003] Currently, with the continuous development of artificial intelligence (AI) technology, it has been applied to various industries. For example, in medical settings, AI technology can be used for information detection to obtain test results, which doctors can then use to make appropriate clinical decisions. For instance, after acquiring medical images of a patient's body part, AI technology can be used to process the image to detect any abnormalities in that area, allowing doctors to make suitable clinical decisions based on the results.

[0004] However, the detection results obtained through related artificial intelligence detection schemes are usually of poor accuracy. Summary of the Invention

[0005] In view of the above, embodiments of this disclosure provide an image processing method, a computer-aided diagnostic method for fatty liver, an apparatus, a system, a computer storage medium, and a computer program product to at least partially solve the above problems.

[0006] According to a first aspect of the present disclosure, an image processing method is provided, comprising:

[0007] Acquire medical images and extract features from the medical images to obtain deep image features;

[0008] The deep image features are encoded to obtain image texture features; wherein, the image texture features characterize the pixel value distribution of each pixel in the medical image.

[0009] By fusing the deep image features and the image texture features, the visual features corresponding to the medical image are obtained;

[0010] Based on the visual features, the detection result is predicted to obtain the predicted detection result.

[0011] According to a second aspect of the present disclosure, a computer-aided diagnostic method for fatty liver is provided, comprising:

[0012] Acquire medical CT images of the liver region of the diagnostic subject, and extract features from the medical CT images to obtain deep image features;

[0013] The deep image features are encoded to obtain image texture features; wherein, the image texture features characterize the pixel value distribution of each pixel in the medical CT image.

[0014] By fusing the deep image features and the image texture features, the visual features corresponding to the medical CT image are obtained;

[0015] Based on the visual features, the diagnostic result is predicted to obtain the fatty liver diagnosis result of the subject.

[0016] According to a third aspect of the present disclosure, a computer-aided diagnostic method for fatty liver is provided, applied to a client device, the method comprising:

[0017] A medical CT image of the liver region of the diagnostic subject is acquired and sent to a server device. The server device extracts the deep image features and image texture features of the medical CT image, fuses the deep image features and image texture features to obtain visual features, and obtains a diagnosis of fatty liver based on the visual features.

[0018] Receive fatty liver diagnosis results returned by the server device;

[0019] The fatty liver diagnosis results are displayed on the screen.

[0020] According to a fourth aspect of the present disclosure, a computer-aided diagnostic method for fatty liver is provided, applied to a server device, the method comprising:

[0021] Receive medical CT images of the liver region of the diagnostic subject sent by the client device, and obtain the deep image features and image texture features of the medical CT images;

[0022] By fusing the deep image features and the image texture features, visual features are obtained;

[0023] Based on the visual features, the diagnostic result is predicted to obtain the fatty liver diagnosis result of the diagnostic subject.

[0024] The fatty liver diagnosis result is returned to the client device so that it can be displayed on the client device's screen.

[0025] According to a fifth aspect of the present disclosure, a client device is provided, comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus; the memory is used to store at least one executable instruction, the executable instruction causing the processor to perform an operation corresponding to the method described in the third aspect.

[0026] According to a sixth aspect of the present disclosure, a server device is provided, comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus; the memory is used to store at least one executable instruction, the executable instruction causing the processor to perform an operation corresponding to the method described in the fourth aspect.

[0027] According to a seventh aspect of the present disclosure, a computer-aided diagnostic system for fatty liver is provided, comprising: a client device as described in the fifth aspect above, and a server device as described in the sixth aspect above.

[0028] According to an eighth aspect of the present disclosure, a computer storage medium is provided that stores a computer program thereon, which, when executed by a processor, implements the method as described in any one of the first to fourth aspects.

[0029] The image processing scheme provided in this embodiment extracts deep image features from medical images to obtain deep image features that can characterize image details. Then, it extracts texture features from the extracted deep image features to obtain image texture features that characterize the distribution of pixel values ​​in the image. This makes the information contained in the final visual features richer. Then, based on the visual features containing rich information, the detection result prediction can be performed to predict more accurate detection results. Attached Figure Description

[0030] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings.

[0031] Figure 1 is a flowchart of the steps of an image processing method according to an embodiment of the present disclosure;

[0032] Figure 2 is a schematic diagram of the visual image feature extraction process in a three-dimensional image scene;

[0033] Figure 3 is a flowchart of the modal alignment process;

[0034] Figure 4 is a schematic diagram of the modal alignment mask MA;

[0035] Figure 5 is a flowchart of the missing information processing;

[0036] Figure 6 is a schematic diagram of the image processing flow in the scenario of fatty liver diagnosis;

[0037] Figure 7 is a schematic diagram of the missing-aware modal alignment (MAMA) adapter;

[0038] Figure 8 is a schematic diagram of an exemplary computer-aided diagnostic system for fatty liver applicable to the embodiments of the present disclosure.

[0039] Figure 9 is a flowchart of a computer-aided diagnostic method for fatty liver according to Embodiment 2 of this disclosure;

[0040] Figure 10 is a flowchart of a computer-aided diagnostic method for fatty liver according to Embodiment 3 of this disclosure;

[0041] Figure 11 is a structural block diagram of an image processing apparatus according to Embodiment 4 of the present disclosure;

[0042] Figure 12 is a structural block diagram of a computer-aided diagnostic device for fatty liver according to Embodiment 5 of the present disclosure;

[0043] Figure 13 is a structural block diagram of a computer-aided diagnostic device for fatty liver according to Embodiment Six of this disclosure;

[0044] Figure 14 is a schematic diagram of a client device according to Embodiment 7 of the present disclosure;

[0045] Figure 15 is a schematic diagram of the structure of a server device according to Embodiment 8 of this disclosure. Detailed Implementation

[0046] To enable those skilled in the art to better understand the technical solutions in the embodiments of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art should fall within the protection scope of this disclosure.

[0047] The specific implementation of the embodiments of this disclosure will be further described below with reference to the accompanying drawings.

[0048] Example 1

[0049] Referring to Figure 1, Figure 1 is a flowchart of an image processing method according to Embodiment 1 of the present disclosure. The image processing method provided by this embodiment can be executed by any suitable device with image processing capabilities. Specifically, the image processing method provided by this embodiment may include the following steps:

[0050] Step 102: Acquire medical images and extract features from them to obtain deep image features.

[0051] Specifically, the medical imaging images in this embodiment can be images obtained by performing a preset detection on a certain part of a medical examination subject. In this embodiment, there are no limitations on the detection content, or the form and content of the imaging images; these can be customized according to actual circumstances. For example, the medical imaging images can be CT images obtained from CT (Computed Tomography) detection of a certain body part of a medical examination subject (such as a person), X-ray images obtained from X-ray detection, MRI (Magnetic Resonance Imaging) images obtained from MRI (Magnetic Resonance Imaging), and so on.

[0052] After acquiring the medical image, feature extraction can be performed to obtain deep image features that reflect the image's detailed information. In this embodiment, the specific method used for deep feature extraction is not limited; it can be customized according to actual conditions. For example, a convolutional neural network (CNN) or a residual neural network (ResNet) model can be used to obtain the aforementioned deep image features.

[0053] Step 104: Encode the deep image features to obtain image texture features; wherein, the image texture features characterize the pixel value distribution of each pixel in the medical image.

[0054] Specifically, image texture features can characterize the pixel value distribution of each pixel in the medical image obtained in step 102.

[0055] Step 106: Fuse deep image features and image texture features to obtain the visual features corresponding to the medical image.

[0056] Specifically, after obtaining deep image features that reflect image detail information through step 102, and obtaining image texture features that characterize the pixel value distribution of pixels in the image through step 104, the visual features that combine image detail information and pixel value distribution information can be obtained through the feature fusion operation in step 106.

[0057] In this embodiment of the disclosure, the extracted deep image features and image texture features can be fused in any suitable manner. For example, the deep image features and image texture features can be added together to obtain a fused visual image; or the weights of the deep image features and image texture features can be set separately first, and then the two features can be weighted and summed to obtain a fused visual image, and so on.

[0058] Step 108: Predict the detection result based on visual features to obtain the predicted detection result.

[0059] Specifically, the predicted detection results in this embodiment can characterize the likelihood of adverse events occurring in the medical subject. The predicted detection results can include: the adverse event and the probability of its occurrence. For example, in a heart detection scenario, a CT image of the heart of the subject can be acquired. After steps 102-108, adverse events in the heart of the subject and the probability of such adverse events can be predicted. In a liver detection scenario, a CT image of the liver of the subject can be acquired. After steps 102-108, adverse events in the liver of the subject and the probability of such adverse events can be predicted.

[0060] Furthermore, in some scenarios, there may be multiple types of adverse events. In this case, the predicted detection results can include: various adverse events, and the probability of each adverse event occurring. For example, in the scenario of liver detection, liver adverse events may include: healthy, mild fatty liver, moderate fatty liver, and severe fatty liver. After obtaining the CT image of the liver of the test subject, the probability of the test subject having a healthy liver, the probability of having mild fatty liver, the probability of having moderate fatty liver, and the probability of having severe fatty liver can be predicted through the above steps 102-108.

[0061] In this embodiment of the disclosure, any suitable classification prediction algorithm can be used to predict the detection result based on visual features, thereby obtaining the predicted detection result.

[0062] The image processing method provided in this embodiment extracts deep image features from medical images to obtain deep image features that can characterize image details. Then, it extracts texture features from the extracted deep image features to obtain image texture features that characterize the distribution of pixel values ​​in the image. This makes the information contained in the final visual features richer. Then, based on the visual features containing rich information, the detection result prediction can be performed to predict more accurate detection results.

[0063] Optionally, in some embodiments, the medical image obtained in step 102 above can be a three-dimensional image. In this case, the process of extracting features from the medical image to obtain deep image features may include:

[0064] The medical image is segmented to obtain multiple three-dimensional image blocks; features are extracted from each three-dimensional image block to obtain the deep image block features corresponding to each three-dimensional image block.

[0065] Encoding deep image features yields image texture features, including:

[0066] Each deep image block is encoded to obtain the image block texture features corresponding to each 3D image block;

[0067] By fusing deep image features and image texture features, visual features are obtained, including:

[0068] The depth image block features and image block texture features corresponding to the same 3D image block are fused to obtain the visual features corresponding to the medical image.

[0069] Specifically, three-dimensional medical images are usually large in size and contain a lot of information. If feature extraction is performed on the entire image, it may put pressure on the memory of the computing device, and the computing efficiency will also be low.

[0070] In the embodiments described above, for large-sized three-dimensional medical images, during feature extraction, the large image is first divided into multiple smaller three-dimensional image blocks. Depth feature extraction and texture feature extraction operations are then performed on each three-dimensional image block. Finally, the deep image features and texture features of the same three-dimensional image block are fused to obtain the visual features corresponding to the three-dimensional medical image. This conversion of large image processing operations into smaller image block processing operations has the following advantages: reduced memory pressure on computing devices; parallel execution of feature extraction operations for each three-dimensional image block using multi-core processors in the computing device, thus improving operational efficiency; three-dimensional image block processing is more conducive to capturing local features in the image, which may be ignored in overall image analysis; if feature extraction of a certain three-dimensional image block fails or is affected by noise, the features of other three-dimensional image blocks can still provide useful information, thus exhibiting high reliability.

[0071] Referring to Figure 2, which is a schematic diagram of the visual image feature extraction process in a 3D image scene, the following explanation of the visual feature acquisition process is provided in conjunction with Figure 2:

[0072] After acquiring a three-dimensional medical image x with dimensions D×W×H.im Then, based on the computing performance of the computing device, the image x can be... im The image is divided into multiple 3D image blocks, where the i-th 3D image block can be denoted as... Then, each 3D image block is input into a preset block encoder (exemplarily, a 3D convolutional neural network model 3D CNN in Figure 2) to extract deep features, thereby obtaining the deep image block features corresponding to each 3D image block. Where n is a natural number, ranging from 1 to N. p N p For natural numbers greater than 1; layer image patch features Input a preset texture encoder to obtain the corresponding image patch texture features e. tex Then, for the same 3D image patch, the depth image patch features are analyzed. and image block texture features e tex By performing fusion (such as feature addition), the corresponding visual features can be obtained. Furthermore, as a data augmentation technique, it is used to obtain visual features. Afterwards, a random zero-mask operation can be performed on it to obtain the masked visual features, which can then be used to predict the detection results.

[0073] In this embodiment of the disclosure, any suitable texture encoder can be used to encode each deep image block to obtain the corresponding image texture features. For example, a texture encoder can be represented as:

[0074] in,() T Indicates the transpose operation; {d k ∈R C} k＝1:K d represents a pre-trained encoded dictionary containing K encoded words. k Indicates the k-th encoded word; {s k ∈R C} k＝1:K S represents the K scaling factors pre-trained. k This represents the k-th scaling factor.

[0075] Optionally, in some embodiments, the process of predicting detection results based on visual features to obtain predicted detection results may include:

[0076] Obtain biometric information that matches medical images, and extract features from the biometric information to obtain biometric features;

[0077] Obtain the pre-trained modal alignment cue vector, and perform modal alignment processing on visual features and biological features based on the modal alignment cue vector to obtain aligned features;

[0078] Based on the attention mechanism, feature interaction processing is performed on each feature in the alignment feature to obtain the interaction result. The detection result is then predicted based on the interaction result to obtain the predicted detection result.

[0079] Specifically, biometric information matching medical imaging images can be other clinical information that matches those images. For example, the medical imaging image can be a CT image of a predetermined site on the subject, and the corresponding biometric information can include blood test (e.g., complete blood count) information obtained through blood tests, and / or biomarker information in the predetermined site on the subject, etc. A biomarker is a measurable biological characteristic of an organism that can serve as an indicator, predictor, or assessment metric for disease, physiological, or pathological processes. Biomarkers can be characteristics of molecules, cells, tissues, organs, or the entire organism, and can be measured quantitatively or qualitatively using various detection methods.

[0080] After obtaining biological detection information that matches the medical image, feature extraction can be performed on the information to obtain biological features. In this embodiment, the specific extraction method used for biological feature extraction is not limited and can be customized according to the actual situation.

[0081] In this embodiment, the visual features are extracted from images, while the biometric features are extracted from biometric detection information. That is, visual features and biometric features belong to two different modalities. Considering the heterogeneity between the two modalities, if subsequent feature fusion and interaction operations are performed immediately after acquiring the visual and biometric features, the final prediction result will not be accurate enough. Therefore, in this embodiment, after acquiring the features of the two different modalities, a modality alignment cue vector is introduced. Based on this vector, modality alignment processing is performed on the visual and biometric features to obtain aligned features, that is, feature representations under the same preset feature space.

[0082] Specifically, refer to Figure 3, which is a schematic flowchart of the modal alignment process. The following explanation of the modal alignment process, in conjunction with Figure 3, will cover the following steps:

[0083] The modality alignment cue vector is concatenated, and visual and biological features are modally aligned to obtain the first concatenated feature; based on the first concatenated feature, the following is obtained:<Q、K、V> Triple vectors; self-attention calculation is performed based on the triple vectors to obtain alignment features.

[0084] Specifically, the self-attention calculation process described above may include: performing matrix multiplication on the Q and K vectors in the triplet vectors to obtain a first calculation result; scaling the first calculation result using a pre-trained scaling factor; and applying a pre-set modality alignment mask M. A The scaling result is masked to obtain a first mask result; the first mask result is normalized to obtain a first self-attention feature vector; the first self-attention feature vector is multiplied by the V vector to obtain the alignment feature. In this embodiment, for the modal alignment mask M... A There are no restrictions on the specific settings; you can customize them according to your actual needs.

[0085] See Figure 4, which shows the modal alignment mask M. A A schematic diagram, where e(P) A ) represents the modality alignment cue vector, and e(ALT) and e(TG) both represent biometric vectors. In Figure 4, the dark-colored areas are the regions to be masked, and the light-colored areas are the regions that do not require masking. When masking the scaling results, it is based on M... A Specifically, this was done by comparing the scaling results with the modal alignment mask M. A The scaling result will be compared with M. A The element value of the element at the same position in the dark element is set to negative infinity, while the element value at other positions remains unchanged.

[0086] In the embodiments described above, the heterogeneity between visual and biological modalities is taken into account. After acquiring visual and biological features, a modality alignment cue vector is introduced. Based on this vector, the visual and biological features are aligned to obtain aligned features, i.e., feature representations under the same preset feature space. Subsequently, feature fusion and interactive operations are performed based on the aligned features, which can effectively improve the accuracy of the prediction results.

[0087] Optionally, in some embodiments, before performing feature interaction processing on the features in the alignment features based on the attention mechanism to obtain the interaction result, the method further includes:

[0088] Obtain the pre-trained missing information cue vector, and perform self-attention calculation on each feature in the alignment feature based on the missing information cue vector to obtain the missing information alignment feature;

[0089] Feature interaction processing is performed on each feature in the alignment feature to obtain the interaction result, including:

[0090] Feature interaction processing is performed on each feature in the missing-aware alignment feature to obtain the interaction result.

[0091] Specifically, in real-world scenarios, for various reasons, it may be impossible to obtain biometric information that matches the medical image, or even if matching biometric information is obtained, it may be missing certain biometric features. In the embodiments described above, a missing feature awareness vector is introduced. Based on this vector, the missing parts of the aligned features can be masked, allowing subsequent feature interactions based on attention mechanisms to focus on the unmissing features while ignoring the missing features. Thus, even with missing biometric information, a relatively accurate prediction result can still be obtained.

[0092] Referring to Figure 5, which is a flowchart illustrating the missing information processing, the following explanation of the missing information processing process, in conjunction with Figure 5, includes the following steps:

[0093] The missing information cue vector and alignment features are concatenated to obtain the second concatenated feature; based on the second concatenated feature, the following is obtained:<Q、K、V> Triple vectors; based on the above triple vectors, self-attention calculation is performed to obtain missing-aware alignment features.

[0094] Specifically, the self-attention computation process described above may include: performing matrix multiplication on the Q and K vectors in the triplet vector to obtain a second operation result; scaling the second operation result using a pre-trained scaling factor; and applying a pre-defined missing information mask M. M The scaling result is masked to obtain a second mask result; the second mask result is normalized to obtain a second self-attention feature vector; the second self-attention feature vector is multiplied by the V vector to obtain the missing-aware alignment feature.

[0095] Optionally, in some embodiments, the image processing can be performed using a pre-trained prediction model. Specifically, the process of extracting features from medical images to obtain deep image features may include:

[0096] The medical image is input into the pre-trained prediction model, and the convolutional network module in the prediction model extracts features from the medical image to obtain deep image features.

[0097] The process of encoding deep image features to obtain image texture features can include:

[0098] The image texture features are obtained by encoding deep image features through the texture encoding module in the prediction model;

[0099] The process of predicting detection results based on visual features can include:

[0100] The attention module in the prediction model performs feature interaction processing on each feature vector in the visual features to obtain output features, and the prediction detection results are obtained based on the output features.

[0101] Specifically, in this embodiment of the disclosure, a prediction model comprising multiple different modules can be used to perform corresponding image processing operations on the acquired medical image to obtain the final prediction detection result. After acquiring the medical image, the image is input into the trained prediction model, and the model can output the prediction detection result, making the operation relatively simple. Furthermore, since the prediction model is pre-trained and possesses corresponding prior prediction knowledge, the output prediction detection result also has high accuracy.

[0102] Furthermore, in some of these embodiments, the prediction model may also include: a biological encoder;

[0103] Image processing methods also include:

[0104] Biological detection information that matches medical images is obtained, and biological features are extracted from the biological detection information through the biological encoder in the prediction model.

[0105] The attention module in the prediction model performs feature interaction processing on the feature vectors in the visual features to obtain output features, and the prediction and detection results are obtained based on the output features. This can include:

[0106] The attention module in the prediction model is used to obtain the pre-trained modal alignment cue vector, and the visual features and the biometric features are modally aligned based on the modal alignment cue vector to obtain alignment features. Based on the attention mechanism, feature interaction processing is performed on each feature in the alignment features to obtain interaction results. The detection results are then predicted based on the interaction results to obtain the predicted detection results.

[0107] Furthermore, in some embodiments, before the attention module in the prediction model performs feature interaction processing on each feature in the alignment features based on the attention mechanism to obtain the interaction result, it can also obtain a pre-trained missing information cue vector, and perform self-attention calculation processing on each feature in the alignment features based on the missing information cue vector to obtain missing information alignment features; then, based on the attention mechanism, it performs feature interaction processing on each feature in the missing information alignment features to obtain the interaction result.

[0108] Optionally, in some embodiments, the training process of the above-described prediction model may include:

[0109] Acquire training sample data; the training sample data includes standard sample pairs and predicted sample pairs; the standard sample pairs include: the first image sample and the corresponding standard label; the predicted sample pairs include: the second image sample and the corresponding predicted label;

[0110] The first image sample is input into the prediction model, and the prediction result of the first sample is obtained through the prediction model; the second image sample is input into the prediction model, and the prediction result of the second sample is obtained through the prediction model.

[0111] Based on the prediction results of the first sample and the standard label, calculate the first loss value; and based on the prediction results of the second sample and the estimated label, calculate the second loss value.

[0112] The first loss value and the second loss value are combined to obtain the combined loss value;

[0113] The model parameters are adjusted based on the fusion loss value to obtain the trained prediction model.

[0114] Specifically, the standard label can be the true label corresponding to the image sample, while the predicted label can be the predicted label obtained through other prediction methods. For example, in the case of fatty liver detection, the standard label can be the label confirmed by biopsy testing, while the predicted label can be the label obtained based on information such as radiological diagnostic reports, or the label obtained through other prediction models.

[0115] For image samples (whether the first image sample or the second image sample), the specific process of obtaining the sample prediction result through the prediction model can include:

[0116] Image samples are input into the prediction model, and deep image sample features are obtained through the convolutional network module in the prediction model; texture features of image samples are obtained through the texture encoding module; and visual sample features are obtained by fusing the deep image sample features and the image sample texture features.

[0117] Add initial classification word vectors and initial distillation word vectors to the visual sample features to obtain the visual sample features to be interacted with.

[0118] The visual sample features to be interacted with are input into the attention module in the prediction model to obtain the output sample features. The output sample features include the output classification word vector corresponding to the initial classification word vector, and the output distillation word vector corresponding to the initial distillation word vector.

[0119] For the first image sample, that is, for the image sample with standard labels, the prediction result can be obtained based on the output classification word vector, and the first loss value can be obtained based on the prediction result and the corresponding standard label.

[0120] For the second image sample, that is, for the image sample with the estimated label, the prediction result can be obtained based on the output distilled word vector, and the second loss value can be obtained based on the prediction result and the corresponding estimated label.

[0121] For standard sample pairs and predicted sample pairs located in the same training batch, the first loss value and the second loss value can be fused to obtain a fused loss value. Then, the parameters of each model in the prediction model can be adjusted according to the fused loss value to obtain the prediction model after training.

[0122] In medical testing scenarios, the number of training samples with real test results that can be collected is usually small. For example, in the case of fatty liver detection, real fatty liver test results are usually obtained through biopsy. However, fatty liver itself is not a serious adverse event. Therefore, the number of people who undergo biopsy to determine whether they have fatty liver is usually small.

[0123] To improve the prediction accuracy of the prediction model, a large number of training sample pairs are usually collected during the model training phase. To address the aforementioned problem, in this embodiment, while collecting training samples with real detection results, data with predicted detection results obtained through prediction methods are also used as training samples, thereby increasing the number of training samples. Therefore, the accuracy of the trained prediction model can be effectively improved.

[0124] Optionally, in some embodiments, the process of obtaining the predicted sample pairs described above may include:

[0125] Obtain candidate image samples and multiple predicted labels corresponding to the candidate image samples; different predicted labels are obtained using different label prediction methods.

[0126] Among the candidate image samples, the image samples with the same predicted labels are determined as the second image sample in the predicted sample pair.

[0127] Specifically, for predicted sample pairs, since the training labels are estimated, there may be cases where the estimated labels are inaccurate or contain noise. Training the model based on inaccurate predicted labels may affect the accuracy of the trained prediction model. Considering the above, in the embodiments of this disclosure, when selecting predicted sample pairs, labels obtained by different methods are compared, and sample pairs with the same predicted labels obtained by different methods are determined as predicted sample pairs for model training. The above process proposes an enhanced label distillation strategy: by comparing predicted labels obtained by different prediction methods and using data with the same predicted labels to participate in model training. Through the above method, the accuracy of the trained prediction model can be further improved.

[0128] In this embodiment of the disclosure, for candidate image samples, any suitable label prediction method can be used to obtain their predicted labels. For example, the label prediction method may include: obtaining the predicted labels based on radiological detection reports (such as CT scan reports, X-ray reports, etc.); it may also include: obtaining the predicted labels using other pre-trained network models, etc.

[0129] Taking the predicted labels obtained from radioactivity detection reports and the predicted labels obtained from other pre-trained network models as examples, the specific formula for calculating the fusion loss value can be:

[0130] Where, N HP Indicates the number of samples in the first image; The prediction result of the first sample corresponding to the i-th first image sample; The standard label corresponds to the i-th first image sample; N NHP Indicates the number of samples in the second image; The prediction result of the second sample corresponding to the i-th second image sample; and Both are the estimated labels corresponding to the i-th second image sample, where, To obtain the estimated label based on the radioactivity test report, The predicted labels are obtained based on other pre-trained network models; L MSE ψ() is the mean squared error loss function; ψ() is the logistic function, an S-shaped curve function; KL() is the KL divergence (Kullback–Leibler divergence, maximum inter-class divergence); This is the first loss value; This is the second loss value; The conditions for calculating the second loss value are as follows: λ is the preset weight value; L is the fusion loss value.

[0131] Referring to Figure 6, which is a schematic diagram of the image processing flow in the fatty liver diagnosis scenario, the image processing flow of this embodiment in the fatty liver diagnosis scenario will be explained below with reference to Figure 6:

[0132] Predictive model training phase:

[0133] The first step is to obtain training sample data. This data contains two different types of sample pairs: the first type is a standard sample pair for which a diagnosis of fatty liver has been confirmed by biopsy, including: the first sample X HP and the corresponding standard label y HP , where the first sample X HP The first type may include three-dimensional CT images of the liver, and optionally, may also include biological detection information matching the aforementioned CT (such as at least one of the following: biomarker information in the liver, blood test information, etc.); the second type: standard sample pairs for determining fatty liver diagnosis results through radiological diagnostic reports and pre-trained models, without biopsy testing, including: second sample X NHP And, the label y determined through radiological diagnostic reports. NHP and the label y determined by the pre-trained model PME , where, with the first sample X HP Similarly, the second sample X NHP It may also include three-dimensional CT images of the liver, and optionally, it may also include biological detection information that matches the CT scan (such as at least one of the following: biomarker information in the liver, blood test information, etc.).

[0134] The second step is feature extraction. After obtaining the training sample data, feature extraction is performed on each sample (including the first sample X). HP Second sample X NHP This allows for feature extraction from both images and biometric detection information. Specifically, as shown in Figure 6, visual features N can be extracted using the image encoder in the prediction model. im The specific processing steps of the image encoder are shown in Figure 2, and will not be elaborated here. Simultaneously, biological features N can be extracted using the biological encoder in the prediction model. bio Then, for visual feature N im and biological characteristics N bio Perform concatenation, and simultaneously add the initial classification word vector t to the concatenated feature vector. cls and the initial distilled word vector t dist This allows us to obtain the initial feature h corresponding to the sample. 0 .

[0135] The third step is feature interaction. This involves processing the initial feature h... 0The input prediction model consists of an encoder (i.e., an attention module) composed of N stacked self-attention units, through which the output feature h is obtained. N , where the output feature h N It contains the output classification word vector FC corresponding to the initial classification word vector. ord And, the initial distilled word vector t dist The corresponding output distilled word vector FC dist As shown in Figure 6, the self-attention unit consists of a missing-aware modal alignment (MAMA) adapter and a Transformer block (including a multi-head self-attention layer and a feedforward network). Referring to Figure 7, the specific structure of the MAMA adapter includes a modal alignment module, a missing-aware module, and a classification layer. For the i-th MAMA adapter, its processing flow is as follows: obtain the pre-trained modal alignment cue vector P. A splicing and aligning the hint vector P A and the output characteristics h of the (i-1)th MAMA adapter l-1 The splicing result is then input into the modal alignment module, which performs the modal alignment processing shown in Figure 3. The result is then normalized by a normalization layer. Finally, the result is compared with the pre-trained missing information cue vector P. M The concatenation is performed, and the concatenation result is input into the missing feature detection module. The missing feature detection module performs missing feature detection processing as shown in Figure 5, and the processing result is normalized by the normalization layer. The concatenation result is added to the above normalization result, and the added result is input into the classification layer and the normalization layer in sequence to finally obtain the output feature h of the i-th MAMA adapter. l .

[0136] The fourth step is to calculate the loss value. Specifically, based on the output features h obtained from the encoder... N The output classification word vector FC in ord Initial distilled word vector t dist The corresponding output distilled word vector FC dist The loss value is calculated based on the sample's label and the corresponding label. Specifically, the loss value calculation can be based on two different branches: a conventional classification branch, and a distillation branch unique to this embodiment. Referring to Figure 6, for the first sample X... HP In the classification branch, based on the classification word vector FC ord and the corresponding standard label y HP The first loss value is calculated; for the second sample X NHP In the distillation branch, the output distillation word vector FC can be used. dist , and, the label y determined through radiological diagnostic reports NHPand the label y determined by the pre-trained model PME The second loss value is calculated; then, the final loss value is obtained by weighted summation of the first loss value obtained through the classification branch and the second loss value obtained through the distillation branch.

[0137] The fifth step involves adjusting the parameters of each module in the prediction model based on the loss value calculated in the fourth step, thereby obtaining the trained prediction model. Specifically, for the MAMA adapter in the encoder module of Figure 6, the trainable parameters within the MAMA adapter can be fine-tuned by freezing the parameters of other modules, resulting in a fine-tuned MAMA adapter.

[0138] Model inference stage:

[0139] The steps in the model inference phase are similar to those in the model training phase, with the following differences: First, the data input to the prediction model changes from training samples to data to be detected. Specifically, this may include CT images of the object to be diagnosed, and further, it may include biometric information of the object to be diagnosed, such as a 3D CT image of the liver of a diagnostic subject, blood test information of the diagnostic subject, and biomarker information of the liver of the diagnostic subject; Second, after obtaining the encoder, the output feature h is obtained. N Then, based on the output classification word vector FC ord The first prediction result is obtained based on the output distilled word vector FC. dist The second prediction result is obtained; then the first and second prediction results are fused (e.g., the probability of the same adverse event occurring is averaged to obtain the probability of the adverse event being sent, etc.) to obtain the final prediction result.

[0140] Referring to Figure 8, which is a schematic diagram of an exemplary computer-aided diagnostic system for fatty liver applicable to the embodiments of the present disclosure, the application scenarios of the computer-aided diagnostic method for fatty liver provided by the embodiments of the present disclosure will be explained first with reference to Figure 1.

[0141] As shown in Figure 8, the system 800 may include a server 802, a communication network 804 and / or one or more user devices (i.e., client devices) 806. The example in Figure 8 shows multiple user devices.

[0142] Server 802 can be any suitable device for assisting in the diagnosis of fatty liver based on medical CT images containing liver tissue, including but not limited to server clusters, cloud computing server clusters, etc. In some embodiments, server 802 can perform any suitable function. For example, in some embodiments, server 802 can be used to receive medical CT images of the liver tissue of the diagnostic subject sent by user equipment 806, perform image processing based on the received medical CT images to obtain a fatty liver diagnosis result for the diagnostic subject, and return the fatty liver diagnosis result to the user equipment 806.

[0143] In some embodiments, the communication network 804 can be any suitable combination of one or more wired and / or wireless networks. For example, the communication network 804 can include any one or more of the following: the Internet, an intranet, a wide area network (WAN), a local area network (LAN), a wireless network, a digital subscriber line (DSL) network, a frame relay network, an asynchronous transfer mode (ATM) network, a virtual private network (VPN), and / or any other suitable communication network. The user equipment 806 can be connected to the communication network 804 via one or more communication links (e.g., communication link 812), and the communication network 804 can be linked to the server 802 via one or more communication links (e.g., communication link 814). The communication link can be any communication link suitable for transmitting data between the user equipment 806 and the server 802, such as a network link, a dial-up link, a wireless link, a hardwired link, any other suitable communication link, or any suitable combination of such links.

[0144] User equipment 806 may include any one or more user devices. In some embodiments, user equipment 806 may include any suitable type of device. For example, in some embodiments, user equipment 806 may include a computer and / or any other suitable type of user device.

[0145] Based on the above system, this disclosure also provides a computer-aided diagnostic method for fatty liver, which will be described below through several embodiments.

[0146] Example 2

[0147] Referring to Figure 9, which is a flowchart illustrating the steps of a computer-aided diagnostic method for fatty liver according to Embodiment 2 of this disclosure, the computer-aided diagnostic method for fatty liver provided in this embodiment can be executed by user equipment 806 in the system shown in Figure 8. Specifically, the computer-aided diagnostic method for fatty liver provided in this embodiment includes the following steps:

[0148] Step 902: Obtain medical CT images of the liver region of the diagnostic subject.

[0149] Specifically, in this step, a CT scan can be performed on the liver of the patient to generate a medical CT image.

[0150] Step 904: The medical CT image is sent to the server device so that the server device can extract the deep image features and image texture features of the medical CT image, fuse the deep image features and image texture features to obtain visual features; and obtain the diagnosis result of fatty liver of the subject based on the visual features.

[0151] Specifically, a fatty liver diagnosis can characterize the likelihood of fatty liver in the liver of the diagnosed individual. That is, the fatty liver diagnosis can include the probability of being healthy and having no fatty liver, as well as the probability of having fatty liver. Furthermore, in practical scenarios, fatty liver can also be graded. For example, fatty liver can be divided into mild fatty liver, moderate fatty liver, and severe fatty liver. In this case, the fatty liver diagnosis could include the probability of being healthy and having no fatty liver, the probability of having mild fatty liver, the probability of having moderate fatty liver, and the probability of having severe fatty liver.

[0152] The specific implementation process of obtaining fatty liver diagnosis results based on medical CT images by the server device will not be described in detail here. For details, please refer to the corresponding steps in the above embodiment one.

[0153] Step 906: Receive the fatty liver diagnosis result returned by the server device.

[0154] Step 908: Display the fatty liver diagnosis results on the display interface.

[0155] The computer-aided diagnosis method for fatty liver provided in this embodiment involves sending a medical CT image of the liver region of the patient to a server device that provides computer-aided diagnosis services for fatty liver. The server device performs deep image feature extraction on the medical CT image to obtain deep image features that characterize image details. Furthermore, texture feature extraction is performed on the extracted deep image features to obtain image texture features that characterize the pixel value distribution in the image, thus enriching the information contained in the final visual features. Then, based on these information-rich visual features, a more accurate fatty liver diagnosis result can be predicted. Finally, the predicted fatty liver diagnosis result is returned to the user device for the user to view.

[0156] Example 3

[0157] Referring to Figure 10, which is a flowchart illustrating the steps of a computer-aided diagnostic method for fatty liver according to Embodiment 3 of this disclosure, the computer-aided diagnostic method for fatty liver provided in this embodiment can be executed by the server 802 in the system shown in Figure 8. Specifically, the computer-aided diagnostic method for fatty liver provided in this embodiment includes the following steps:

[0158] Step 1002: Receive medical CT images of the liver region of the diagnostic subject sent by the client device, and obtain the deep image features and image texture features of the medical CT images.

[0159] Specifically, a CT scan can be performed on the liver of the patient to generate a medical CT image. After generating the image, the client device can send it to the server device for computer-aided diagnosis of fatty liver.

[0160] Step 1004: Fuse deep image features and image texture features to obtain visual features.

[0161] Step 1006: Based on visual features, predict the diagnostic results to obtain the fatty liver diagnosis result of the subject.

[0162] Specifically, a fatty liver diagnosis can characterize the likelihood of fatty liver in the liver of the diagnosed individual. That is, the fatty liver diagnosis can include the probability of being healthy and having no fatty liver, as well as the probability of having fatty liver. Furthermore, in practical scenarios, fatty liver can also be graded. For example, fatty liver can be divided into mild fatty liver, moderate fatty liver, and severe fatty liver. In this case, the fatty liver diagnosis could include the probability of being healthy and having no fatty liver, the probability of having mild fatty liver, the probability of having moderate fatty liver, and the probability of having severe fatty liver.

[0163] The specific implementation process of obtaining fatty liver diagnosis results based on medical CT images by the server device will not be described in detail here. For details, please refer to the corresponding steps in the above embodiment one.

[0164] Step 1008: Return the fatty liver diagnosis result to the client device so that the fatty liver diagnosis result can be displayed through the client device's display interface.

[0165] The computer-aided diagnosis method for fatty liver provided in this embodiment involves a server device receiving a medical CT image of the liver region of the diagnostic subject from a user device. The server device performs deep image feature extraction on the medical CT image to obtain deep image features that characterize image details. Furthermore, it extracts texture features from the extracted deep image features to obtain image texture features that characterize the distribution of pixel values ​​in the image, thus enriching the information contained in the final visual features. Then, based on these information-rich visual features, a more accurate diagnosis of fatty liver can be predicted. Finally, the predicted fatty liver diagnosis result is returned to the user device for the user to view.

[0166] Experimental data and results

[0167] To verify the effectiveness of the computer-aided diagnostic method for fatty liver provided in this disclosure, a prediction model was trained using a dataset, and the model's performance was evaluated. Specifically:

[0168] The dataset includes one internal dataset and two external datasets. The internal dataset contains 680 pathologically confirmed subjects (203 healthy, 250 with mild fatty liver, 138 with moderate fatty liver, and 89 with severe fatty liver) and 1,103 medically confirmed subjects (438 healthy, 307 with mild fatty liver, 112 with moderate fatty liver, and 246 with severe fatty liver). The external datasets consist of the public dataset UNIFESP, which includes 226 subjects who underwent chest and abdominal scans. One dataset consists of 79 healthy individuals and 147 individuals with mild fatty liver disease. Another external dataset is an instance dataset of 13,948 subjects collected from three different clinical sites, specifically: subjects collected through physical examinations (6,051, with a prevalence of steatosis of 17.5%), subjects collected through inpatient departments (3,451, with a prevalence of steatosis of 5.9%), and subjects collected through outpatient departments (4,446, with a prevalence of steatosis of 12.7%).

[0169] In the context of computer-aided diagnosis of fatty liver, the diagnostic results can be set to include the following three types: mild fatty liver, moderate fatty liver, and severe fatty liver.

[0170] During the model training phase, the aforementioned internal dataset was split into a training set (accounting for 70% of the total internal dataset) and a validation set (accounting for 30% of the total internal dataset). The model was trained based on the training set and validated based on the training set.

[0171] After obtaining the trained prediction model, this embodiment of the present disclosure also compares the performance of the computer-aided diagnosis method for fatty liver provided in this embodiment of the present disclosure with other related processing methods based on the aforementioned dataset. See Table 1 below, which shows the diagnostic performance of different diagnostic methods on multiple datasets.

[0172] Table 1

[0173] Among them, MICE (Multiple Imputation by Chained Equation), KNN Impute (Imputation by K-nearest neighbor), MMFormer, TagAssisted, and MultiPrompt, listed in the table above, are all commonly used computer-aided diagnostic methods for fatty liver. When evaluating model performance, AUC (Area Under Curve) and BACC (Balanced Accuracy) were used as performance metrics.

[0174] As shown in Table 1, the multimodal diagnostic method generally outperforms the single-modal method. Furthermore, regardless of whether the dataset is the aforementioned internal dataset, UNIFESP, or any of the instance datasets, the computer-aided diagnostic method for fatty liver provided in this disclosure achieves higher AUC and BACC compared to existing multimodal fatty liver diagnostic schemes. In other words, the scheme provided in this disclosure has better detection performance compared to other methods.

[0175] Furthermore, to compare the effects of different embodiments of this disclosure, an internal ablation experiment was conducted on the computer-aided diagnosis scheme for fatty liver in the fatty liver diagnosis scenario shown in Figure 6. The specific experimental results are shown in Table 2 below:

[0176] Table 2

[0177] As shown in Table 2 above, the introduction of the label distillation strategy can effectively improve the predictive performance of the prediction model. Specifically, it increases the mean BACC by 2% and the mean AUC by 1.4% overall. The random zero-mask strategy also increases the mean AUC, but the increase is not significant. When removing biometric information and the MAMA adapter, using image-based single-modality detection reduces the mean BACC to 85.3% and the mean AUC to 88.9% of the complete scheme. Compared with the multimodal approach, the mean BACC decreases by approximately 0.8%, while the mean AUC decreases by approximately 1%. Furthermore, introducing texture features during the detection process is important for improving the diagnostic performance of mild and moderate fatty liver. Specifically, for mild fatty liver detection, the mean BACC can increase by approximately 3%, and for moderate fatty liver diagnosis, the mean BACC can increase by approximately 1.8%.

[0178] Example 4

[0179] Figure 11 is a structural block diagram of an image generation apparatus according to Embodiment 4 of this disclosure. The image generation apparatus provided in this embodiment includes:

[0180] The deep feature extraction unit 1102 is used to acquire medical image images and extract features from the medical image images to obtain deep image features;

[0181] The texture feature extraction unit 1104 is used to encode deep image features to obtain image texture features; wherein, the image texture features characterize the pixel value distribution of each pixel in the medical image.

[0182] The feature fusion unit 1106 is used to fuse deep image features and image texture features to obtain the visual features corresponding to the medical image;

[0183] The prediction unit 1108 is used to predict the detection result based on visual features to obtain the predicted detection result.

[0184] Optionally, in some embodiments, the medical image is a three-dimensional image, and the deep feature extraction unit 1102, when performing the step of extracting features from the medical image to obtain deep image features, is specifically used for:

[0185] The medical image is segmented to obtain multiple three-dimensional image blocks; features are extracted from each three-dimensional image block to obtain the deep image block features corresponding to each three-dimensional image block.

[0186] Texture feature extraction unit 1104 is specifically used for:

[0187] Each deep image block is encoded to obtain the image block texture features corresponding to each 3D image block;

[0188] Feature fusion unit 1106 is specifically used for:

[0189] The depth image block features and image block texture features corresponding to the same 3D image block are fused to obtain the visual features corresponding to the medical image.

[0190] Optionally, in some embodiments, the prediction unit 1108 is specifically used for:

[0191] Obtain biometric information that matches medical images, and extract features from the biometric information to obtain biometric features;

[0192] Obtain the pre-trained modal alignment cue vector, and perform modal alignment processing on visual features and biological features based on the modal alignment cue vector to obtain aligned features;

[0193] Based on the attention mechanism, feature interaction processing is performed on each feature in the alignment feature to obtain the interaction result. The detection result is then predicted based on the interaction result to obtain the predicted detection result.

[0194] Optionally, in some embodiments, the prediction unit 1108, before performing feature interaction processing on each feature in the aligned features based on an attention mechanism to obtain the interaction result, is further configured to:

[0195] Obtain the pre-trained missing information cue vector, and perform self-attention calculation on each feature in the alignment feature based on the missing information cue vector to obtain the missing information alignment feature;

[0196] Prediction unit 1108, when performing the step of performing feature interaction processing on each feature in the aligned features to obtain the interaction result, is specifically used for:

[0197] Feature interaction processing is performed on each feature in the missing-aware alignment feature to obtain the interaction result.

[0198] Optionally, in some embodiments, when performing the step of extracting features from medical images to obtain deep image features, the deep feature extraction unit 1102 is specifically used for:

[0199] The medical image is input into the pre-trained prediction model, and the convolutional network module in the prediction model extracts features from the medical image to obtain deep image features.

[0200] The texture feature extraction unit 1104 is specifically used to: encode deep image features through the texture coding module in the prediction model to obtain image texture features;

[0201] The prediction unit 1108 is specifically used to: perform feature interaction processing on each feature vector in the visual features through the attention module in the prediction model to obtain output features, and obtain prediction detection results based on the output features.

[0202] Optionally, in some embodiments, the image processing apparatus further includes:

[0203] The model training module is used to acquire training sample data. The training sample data includes standard sample pairs and predicted sample pairs. The standard sample pairs include: a first image sample and its corresponding standard label; the predicted sample pairs include: a second image sample and its corresponding predicted label. The first image sample is input into the prediction model, and the prediction model obtains the prediction result of the first sample. The second image sample is input into the prediction model, and the prediction model obtains the prediction result of the second sample. Based on the prediction result of the first sample and the standard label, a first loss value is calculated. Based on the prediction result of the second sample and the predicted label, a second loss value is calculated. The first loss value and the second loss value are fused to obtain a fused loss value. The model parameters of the prediction model are adjusted according to the fused loss value to obtain the trained prediction model.

[0204] The image processing apparatus of this disclosure is used to implement the method in the foregoing method embodiment one, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here. Furthermore, the functional implementation of each unit in the image processing apparatus of this disclosure can be referred to the description of the corresponding part in the foregoing method embodiment, which will also not be repeated here.

[0205] Example 5

[0206] Figure 12 is a structural block diagram of a computer-aided diagnostic device for fatty liver according to Embodiment 5 of this disclosure. The computer-aided diagnostic device for fatty liver provided in this embodiment includes:

[0207] The image acquisition unit 1202 is used to acquire medical CT images of the liver region of the diagnostic subject;

[0208] The image sending unit 1204 is used to send medical CT images to the server device, so that the server device can extract deep image features and image texture features of the medical CT images, fuse the deep image features and image texture features to obtain visual features; and obtain the diagnosis result of fatty liver of the diagnostic object based on the visual features.

[0209] The result receiving unit 1206 is used to receive the fatty liver diagnosis result of the diagnostic object returned by the server device;

[0210] The result display unit 1208 is used to display the fatty liver diagnosis results of the diagnostic subject in the display interface.

[0211] The computer-aided diagnostic device for fatty liver in this embodiment is used to implement the method in the aforementioned method embodiment two, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here. Furthermore, the functional implementation of each unit in the computer-aided diagnostic device for fatty liver in this embodiment can be referred to the description of the corresponding part in the aforementioned method embodiment, which will also not be repeated here.

[0212] Example 6

[0213] Figure 13 is a structural block diagram of a computer-aided diagnostic device for fatty liver according to Embodiment Six of this disclosure.

[0214] The image receiving unit 1302 is used to receive medical CT images of the liver region of the diagnostic object sent by the client device, and to acquire the deep image features and image texture features of the medical CT images.

[0215] Visual feature acquisition unit 1304 is used to fuse deep image features and image texture features to obtain visual features;

[0216] The result is unit 1306, which is used to predict the diagnosis result based on visual features and obtain the fatty liver diagnosis result of the subject.

[0217] The result return module 1308 is used to return the fatty liver diagnosis result of the diagnosed object to the client device so that the fatty liver diagnosis result of the diagnosed object can be displayed through the display interface of the client device.

[0218] The computer-aided diagnostic device for fatty liver in this embodiment is used to implement the method in the aforementioned method embodiment three, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here. Furthermore, the functional implementation of each unit in the computer-aided diagnostic device for fatty liver in this embodiment can be referred to the description of the corresponding part in the aforementioned method embodiment, which will also not be repeated here.

[0219] Example 7

[0220] Referring to FIG14, a schematic diagram of a client device according to Embodiment 7 of the present disclosure is shown. The specific embodiments of the present disclosure do not limit the specific implementation of the client device.

[0221] As shown in Figure 14, the electronic device may include: a processor 1402, a communications interface 1404, a memory 1406, and a communications bus 1408.

[0222] in:

[0223] The processor 1402, communication interface 1404, and memory 1406 communicate with each other via communication bus 1408.

[0224] Communication interface 1404 is used to communicate with other electronic devices or servers.

[0225] The processor 1402 is used to execute program 1410, specifically to perform the relevant steps in the above method embodiments.

[0226] Specifically, program 1410 may include program code that includes computer operation instructions.

[0227] The processor 1402 may be a CPU, an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present disclosure. The smart device may include one or more processors of the same type, such as one or more CPUs; or it may include processors of different types, such as one or more CPUs and one or more ASICs.

[0228] Memory 1406 is used to store program 1410. Memory 1406 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0229] Specifically, program 1410 can be used to cause processor 1402 to execute the relevant steps in the above method embodiments.

[0230] The specific implementation of each step in program 1410 can be found in the corresponding descriptions in the above method embodiments, and will not be repeated here. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the device and module described above can be referred to the corresponding process descriptions in the foregoing method embodiments, and will not be repeated here.

[0231] Example 8

[0232] Referring to Figure 15, a schematic diagram of a server device according to Embodiment 8 of the present disclosure is shown. The specific embodiments of the present disclosure do not limit the specific implementation of the server device.

[0233] As shown in Figure 15, the electronic device may include: a processor 1502, a communications interface 1504, a memory 1506, and a communications bus 1508.

[0234] in:

[0235] The processor 1502, communication interface 1504, and memory 1506 communicate with each other via communication bus 1508.

[0236] Communication interface 1504 is used to communicate with other electronic devices or servers.

[0237] The processor 1502 is used to execute program 1510, which can specifically perform the relevant steps in the above method embodiments.

[0238] Specifically, program 1510 may include program code that includes computer operation instructions.

[0239] The processor 1502 may be a CPU, an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present disclosure. The smart device may include one or more processors of the same type, such as one or more CPUs; or it may include processors of different types, such as one or more CPUs and one or more ASICs.

[0240] Memory 1506 is used to store program 1510. Memory 1506 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0241] Specifically, program 1510 can be used to cause processor 1502 to execute the relevant steps in the above method embodiments.

[0242] The specific implementation of each step in procedure 1510 can be found in the corresponding descriptions in the above method embodiments, and will not be repeated here. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the device and module described above can be referred to the corresponding process descriptions in the foregoing method embodiments, and will not be repeated here.

[0243] This disclosure also provides a computer storage medium storing a computer program thereon, which, when executed by a processor, implements the method described in any of the foregoing method embodiments. The computer storage medium includes, but is not limited to, compact disc read-only memory (CD-ROM), random access memory (RAM), floppy disk, hard disk, or magneto-optical disk.

[0244] This disclosure also provides a computer program product, including computer instructions that instruct a computing device to perform an operation corresponding to any of the methods in the above-described plurality of method embodiments.

[0245] Furthermore, it should be noted that the user-related information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to sample data used for training the model, data used for analysis, stored data, displayed data, etc.) involved in the embodiments of this disclosure are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0246] It should be noted that, depending on the implementation needs, the various components / steps described in the embodiments of this disclosure can be broken down into more components / steps, or two or more components / steps or parts of the operation of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of this disclosure.

[0247] The methods described above according to embodiments of this disclosure can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CD-ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or as computer code downloaded over a network that is originally stored in a remote recording medium or a non-transitory machine-readable medium and will be stored in a local recording medium. Thus, the methods described herein can be processed by software stored on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an ASIC or FPGA). It is understood that the computer, processor, microprocessor controller, or programmable hardware includes storage components (e.g., RAM, ROM, flash memory, etc.) capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods described herein. Furthermore, when a general-purpose computer accesses code used to implement the methods shown herein, the execution of the code transforms the general-purpose computer into a dedicated computer for performing the methods shown herein.

[0248] Those skilled in the art will recognize that the units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments disclosed herein.

[0249] The above embodiments are only used to illustrate the embodiments of this disclosure, and are not intended to limit the embodiments of this disclosure. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the embodiments of this disclosure. Therefore, all equivalent technical solutions also fall within the scope of the embodiments of this disclosure, and the patent protection scope of the embodiments of this disclosure should be defined by the claims.

Claims

1. An image processing method, comprising: Acquire medical images and extract features from the medical images to obtain deep image features; The deep image features are encoded to obtain image texture features; wherein, the image texture features characterize the pixel value distribution of each pixel in the medical image. By fusing the deep image features and the image texture features, the visual features corresponding to the medical image are obtained; Based on the visual features, the detection result is predicted to obtain the predicted detection result.

2. The method according to claim 1, wherein, The medical image is a three-dimensional image; the feature extraction of the medical image to obtain deep image features includes: The medical image is segmented to obtain multiple three-dimensional image blocks; features are extracted from each three-dimensional image block to obtain the deep image block features corresponding to each three-dimensional image block. The encoding process of the deep image features to obtain image texture features includes: Each deep image block is encoded to obtain the image block texture features corresponding to each 3D image block; The fusion of the deep image features and the image texture features to obtain visual features includes: The depth image block features and image block texture features corresponding to the same three-dimensional image block are fused to obtain the visual features corresponding to the medical image.

3. The method according to claim 1 or 2, wherein, The step of predicting the detection result based on the visual features to obtain the predicted detection result includes: Obtain biological detection information that matches the medical image, and extract features from the biological detection information to obtain biological features; Obtain the pre-trained modality alignment cue vector, and perform modality alignment processing on the visual features and the biometric features based on the modality alignment cue vector to obtain alignment features; Based on the attention mechanism, feature interaction processing is performed on each feature in the alignment feature to obtain the interaction result, and the detection result is predicted based on the interaction result to obtain the predicted detection result.

4. The method according to claim 3, wherein, Before performing feature interaction processing on each feature in the alignment features based on the attention mechanism to obtain the interaction result, the method further includes: Obtain the pre-trained missing information cue vector, and perform self-attention calculation on each feature in the alignment feature based on the missing information cue vector to obtain the missing information alignment feature; The features in the alignment features are subjected to feature interaction processing to obtain the interaction result, including: The features in the missing awareness alignment feature are processed by feature interaction to obtain the interaction result.

5. The method according to claim 1, wherein, The step of extracting features from the medical image to obtain deep image features includes: The medical image is input into a pre-trained prediction model, and the convolutional network module in the prediction model is used to extract features from the medical image to obtain deep image features. The encoding process of the deep image features to obtain image texture features includes: The deep image features are encoded using the texture encoding module in the prediction model to obtain image texture features; The step of predicting the detection result based on the visual features to obtain the predicted detection result includes: The attention module in the prediction model performs feature interaction processing on each feature vector in the visual features to obtain output features, and the prediction detection result is obtained based on the output features.

6. The method according to claim 5, wherein, The training process of the prediction model includes: Acquire training sample data; the training sample data includes standard sample pairs and predicted sample pairs; the standard sample pairs include: a first image sample and a corresponding standard label; the predicted sample pairs include: a second image sample and a corresponding predicted label; The first image sample is input into the prediction model, and the prediction model is used to obtain the prediction result of the first sample; the second image sample is input into the prediction model, and the prediction model is used to obtain the prediction result of the second sample. Based on the prediction results of the first sample and the standard label, a first loss value is calculated; and based on the prediction results of the second sample and the estimated label, a second loss value is calculated. The first loss value and the second loss value are combined to obtain the fused loss value; The model parameters of the prediction model are adjusted based on the fusion loss value to obtain the trained prediction model.

7. A computer-aided diagnostic method for fatty liver, comprising: Acquire medical CT images of the liver region of the diagnostic subject, and extract features from the medical CT images to obtain deep image features; The deep image features are encoded to obtain image texture features; wherein, the image texture features characterize the pixel value distribution of each pixel in the medical CT image. By fusing the deep image features and the image texture features, the visual features corresponding to the medical CT image are obtained; Based on the visual features, the diagnostic result is predicted to obtain the fatty liver diagnosis result of the subject.

8. A computer-aided diagnostic method for fatty liver, applied to a client device, the method comprising: A medical CT image of the liver region of the diagnostic subject is acquired and sent to a server device. The server device extracts the deep image features and image texture features of the medical CT image, fuses the deep image features and image texture features to obtain visual features, and obtains a diagnosis of fatty liver based on the visual features. Receive fatty liver diagnosis results returned by the server device; The fatty liver diagnosis results are displayed on the screen.

9. A computer-aided diagnostic method for fatty liver, applied to a server-side device, the method comprising: Receive medical CT images of the liver region of the diagnostic subject sent by the client device, and obtain the deep image features and image texture features of the medical CT images; By fusing the deep image features and the image texture features, visual features are obtained; Based on the visual features, the diagnostic result is predicted to obtain the fatty liver diagnosis result of the diagnostic subject. The fatty liver diagnosis result is returned to the client device so that it can be displayed on the client device's screen.

10. A client device, comprising: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction that causes the processor to perform the operation corresponding to the method as described in claim 8.

11. A server-side device, comprising: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction that causes the processor to perform the operation corresponding to the method as described in claim 9.

12. A computer-aided diagnostic system for fatty liver, comprising: The client device of claim 10 and the server device of claim 11.

13. A computer storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any one of claims 1-9.

14. A computer program product comprising computer instructions that instruct a computing device to perform an operation corresponding to any one of the methods described in claims 1-9.

Images