Training methods for fatty liver grading models, fatty liver grading methods, electronic devices and storage media

Through multi-stage course learning and cross-modal semantic condensation processing, the gradient confusion problem in fatty liver grading was solved, improving the accuracy and generalization ability of the model in classifying mild and moderate fatty liver, thus meeting the needs of clinical applications.

CN122839095BActive Publication Date: 2026-11-10脉得智能科技(无锡)有限公司
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202611317203.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2026-08-28
Publication Date
2026-11-10
Estimated Expiration
2046-08-28

AI Technical Summary

Technical Problem

Existing technologies suffer from gradient confusion in fatty liver grading, resulting in low classification accuracy and insufficient generalization ability of the model at fuzzy decision boundaries such as mild and moderate fatty liver.

Method used

A multi-stage learning strategy, from coarse to fine, is adopted to construct a fatty liver grading model through comparative learning pre-training, five-category global cognition establishment, two-category feature focusing, and four-category fine-grained fine-tuning. Cross-modal semantic condensation and visual cues are used to process ultrasound images and text descriptions to ensure gradient coordination of the model at different granularities.

Benefits of technology

It improves the model's classification accuracy and generalization ability on fuzzy decision boundaries such as mild and moderate fatty liver, avoids gradient confusion, and meets the accuracy and robustness requirements for clinical deployment.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN122839095B_ABST
    Figure CN122839095B_ABST
Patent Text Reader

Abstract

This invention provides a method for training a fatty liver grading model, a fatty liver grading method, an electronic device, and a storage medium, relating to the field of medical image processing technology. A training sample set is obtained, and ultrasound images and text descriptions are input into a dual-tower network for comparative learning to obtain an image encoder. Feature vectors from all images in each case image package are extracted and stacked according to sequence dimensions to obtain a variable-length feature sequence tensor. A multi-instance learning network is trained using five-class labels as supervision signals to obtain a five-class liver model. The classification head is then sequentially replaced with two-dimensional and four-dimensional labels, and progressive supervised training is performed using two-class and four-class labels as supervision signals respectively to obtain a fatty liver grading model. This improves the model's classification accuracy and generalization ability at fuzzy decision boundaries such as mild and moderate fatty liver.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of medical image processing technology, and more specifically, to a method for training a fatty liver grading model, a fatty liver grading method, an electronic device, and a storage medium. Background Technology

[0002] In the field of automatic medical image classification and assisted diagnosis, existing technologies for fine-grained grading of diffuse liver lesions typically employ end-to-end supervised learning strategies. This requires deep convolutional neural networks or visual Transformer models to directly map fine-grained grading results (e.g., mild fatty liver, moderate fatty liver, severe fatty liver) from input ultrasound images within a single training phase. This approach attempts to drive the model to simultaneously complete multiple sub-tasks, such as lesion region localization, pathological feature extraction, and severity determination, in the feature space using a single classification loss function.

[0003] However, the aforementioned one-step fine-grained supervision strategy has optimization flaws in practical clinical applications. From a medical representation perspective, the inter-class variance between adjacent disease levels is extremely small. For example, the difference between mild and moderate fatty liver on ultrasound images is only a slight increase in echo attenuation, and the decision boundary between the two is extremely vague. If fine-grained classification supervision signals are forcibly applied before the network has established a holistic understanding of diffuse lesions (such as fatty liver, cirrhosis, polycystic liver disease, etc.), it will lead to conflicting gradient update directions in the early stages of training. That is, some gradients guide the model to focus on the coarse-grained distinction of "presence or absence of lesions," while other gradients require the model to precisely quantify the fine-grained differences in "lesion severity." These two optimization objectives form an antagonistic relationship in the parameter space, i.e., gradient confusion. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide a method for training a fatty liver grading model, a fatty liver grading method, an electronic device, and a storage medium to at least partially improve the above-mentioned problems.

[0005] To achieve the above objectives, the technical solutions adopted in the embodiments of the present invention are as follows: In a first aspect, embodiments of the present invention provide a method for training a fatty liver grading model, comprising: Obtain a training sample set; the training sample set includes multiple case image packages, each containing multiple ultrasound images of the same case, text descriptions, a five-category label for classifying diffuse lesions, a two-category label for classifying whether it is fatty liver, and a four-category label for classifying the severity of fatty liver. The ultrasound image and its corresponding text description are respectively input into the basic image encoder and the basic text encoder for comparative learning to obtain a preliminarily trained image encoder. Each case image package is input into the pre-trained image encoder to extract the feature vectors of each ultrasound image, and all feature vectors in the same case image package are stacked according to the sequence dimension to obtain a variable-length feature sequence tensor. Based on the variable-length feature sequence tensor and the corresponding five-class labels, a pre-defined basic multi-instance learning network is trained under supervision to obtain a liver five-class classification model. The classification head of the liver five-class classification model is replaced with a two-dimensional classification head, and the two-class classification label is used as a supervision signal for supervised training to obtain a fatty liver two-class classification model. The two-dimensional classifier head of the fatty liver binary classification model is replaced with a four-dimensional classifier head, and the four-dimensional classification labels are used as supervision signals for supervised training to obtain a fatty liver grading model.

[0006] Optionally, the step of obtaining the text description includes: Obtain the original unstructured electronic medical record text corresponding to each case image package, parse the electronic medical record text, and extract image visual feature descriptions and diagnostic opinions; The image visual feature description and the diagnostic opinion are input into a pre-trained large language model to perform cross-modal semantic condensation and obtain the text description.

[0007] Optionally, the step of acquiring multiple ultrasound images from the case image package includes: Remove non-medical UI borders and Doppler modal information from the original ultrasound image, and retain the pure grayscale ultrasound scanning area; Using a pre-trained section classification model, N standard section images for diffuse lesions are selected from the pure grayscale ultrasound scanning area. The pre-trained target detection model is invoked to automatically identify the liver parenchyma and renal cortex regions in each of the standard cross-sectional images, and corresponding bounding boxes are drawn on the original image pixel level. The standard cross-sectional images with bounding boxes are used as ultrasound images in the case image package.

[0008] Optionally, the step of inputting the ultrasound image and its corresponding text description into a base image encoder and a base text encoder respectively for comparative learning to obtain a preliminarily trained image encoder includes: A batch of ultrasound images and their corresponding text descriptions are input into the basic image encoder and the basic text encoder, respectively, to obtain the corresponding number of image feature vectors and text feature vectors; After normalizing the image feature vector and the text feature vector, the cosine similarity between all image feature vectors and text feature vectors within the batch is calculated. Based on the cosine similarity, the image-to-text contrast loss and the text-to-image contrast loss are calculated, and the average of the two is used as the overall contrast loss for this batch. Based on the overall contrast loss, the network weights of the basic image encoder and the basic text encoder are updated using the backpropagation algorithm to obtain the initially trained image encoder.

[0009] Optionally, the step of supervising the training of a preset basic multi-instance learning network based on the variable-length feature sequence tensor and the corresponding five-class classification labels to obtain a liver five-class classification model includes: The variable-length feature sequence tensor corresponding to each of the case image packages is input into a preset basic multi-instance learning network to obtain the classification prediction probability. Freeze all parameters of the image encoder, and independently supervise the training of the basic multi-instance learning network based on the five-class labels and the classification prediction probabilities until the basic multi-instance learning network initially converges. The image encoder is physically spliced ​​with the initially converged basic multi-instance learning network, and the parameters of the last M network layers near the output of the image encoder are unfrozen. A liver five-class classification model is obtained by end-to-end joint supervision training using differential learning rate and the five-class classification labels as supervision signals; wherein the learning rate of the basic multi-instance learning network is higher than the learning rate of the unfrozen part of the image encoder.

[0010] Optionally, replacing the classification head of the liver five-class classification model with a two-dimensional classification head, and performing supervised training using the two-class classification labels as supervisory signals to obtain a fatty liver two-class classification model, includes: Inheriting all the weights of the liver five-class classification model, the classification head of the liver five-class classification model is replaced with a two-dimensional classification head; the two-dimensional classification head is used to distinguish between fatty liver and non-fatty liver. The liver five-class classification model after the classification head is replaced is trained using the binary classification labels as supervision signals. This causes the attention mechanism of the multi-instance learning network in the liver five-class classification model to converge to the fatty liver-specific visual representation, thus obtaining a fatty liver binary classification model.

[0011] Optionally, replacing the two-dimensional classifier head of the fatty liver binary classification model with a four-dimensional classifier head, and using the four-classification labels as supervisory signals for supervised training to obtain a fatty liver grading model, includes: Inheriting all the weights of the fatty liver binary classification model, the two-dimensional classification head of the fatty liver binary classification model is replaced with a four-dimensional classification head; the four dimensions of the four-dimensional classification head correspond to non-fatty liver, mild fatty liver, moderate fatty liver and severe fatty liver, respectively; Using the four-class labels as supervision signals and the focus loss function as the optimization objective, the fatty liver binary classification model after replacing the classification head is trained under supervision to obtain a fatty liver grading model. In the focus loss function, the weight of severe fatty liver is greater than that of moderate fatty liver, greater than that of mild fatty liver, and greater than that of non-fatty liver.

[0012] Secondly, embodiments of the present invention provide a method for grading fatty liver, including: Obtain a package of images of cases to be graded; the package of images of cases to be graded includes multiple ultrasound images of the same case; The image package of cases to be graded is input into the fatty liver grading model to obtain the grading result of the image package of cases to be graded; the fatty liver grading model is trained by the fatty liver grading model training method as described in any of the first aspects.

[0013] Thirdly, embodiments of the present invention provide an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the method described in any of the above-mentioned embodiments.

[0014] Fourthly, embodiments of the present invention provide a storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method described in any of the preceding claims.

[0015] The present invention provides a method for training a fatty liver grading model, a fatty liver grading method, an electronic device, and a storage medium. Through a multi-stage course learning strategy from coarse to fine, the model is forced to perform four-class fine-grained fine-tuning after five-class coarse localization and two-class feature locking. This effectively avoids gradient confusion and pattern collapse caused by direct fine-grained supervision, and improves the model's classification accuracy and generalization ability on fuzzy decision boundaries such as mild and moderate fatty liver.

[0016] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A schematic structural block diagram of an electronic device provided in an embodiment of the present invention; Figure 2 A flowchart illustrating a method for training a fatty liver grading model according to an embodiment of the present invention; Figure 3 This is a schematic diagram of a process for obtaining text descriptions provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of a process for acquiring multiple ultrasound images from a case image package, provided by an embodiment of the present invention. Figure 5 A flowchart illustrating step S22 provided in an embodiment of the present invention; Figure 6 A flowchart illustrating step S24 provided in an embodiment of the present invention; Figure 7 This is a flowchart illustrating a method for grading fatty liver according to an embodiment of the present invention.

[0019] Icons: 100 - Electronic device; 101 - Memory; 102 - Communication interface; 103 - Processor; 104 - Communication bus. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0021] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0022] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this invention, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0023] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0024] In the field of automatic medical image classification and assisted diagnosis, existing technologies for feature extraction and fine-grained grading of diffuse liver lesions are mainly based on traditional single-modal supervised learning techniques using convolutional neural networks or visual Transformers. This approach directly maps a single medical ultrasound image or image sequence to its corresponding discrete category labels (e.g., normal, mild fatty liver, moderate fatty liver, severe fatty liver) for end-to-end training. This type of method directly utilizes image pixel information to fit the classification boundary, resulting in a simple technical architecture and low implementation threshold.

[0025] However, when applying the aforementioned existing technologies directly to the highly visually ambiguous clinical diagnostic scenario of fine-grained grading of diffuse liver lesions (such as fatty liver), these technologies typically employ a one-step target classification strategy (i.e., directly requiring the model to output sub-probabilities such as mild, moderate, and severe). From a medical representation perspective, the inter-class variance between adjacent disease levels is extremely small. If fine-grained classification supervision signals are forcibly applied before the network has established a global understanding of diffuse lesions, it will lead to conflicting gradient update directions in the early stages of training. Some gradients guide the model to focus on the coarse-grained distinction of "presence or absence of lesions," while others require the model to precisely quantify the fine-grained differences in "lesion severity." These two optimization objectives form an antagonistic relationship in the parameter space, i.e., gradient confusion. This suboptimal optimization paradigm causes the network to quickly fall into local optima or mode collapse, and the feature space cannot form a clinically meaningful pathological representation manifold. The classification accuracy for samples bordering between mild and moderate lesions is particularly low, and the generalization ability is significantly reduced, failing to meet the accuracy and robustness standards for high-level medical devices in clinical deployment.

[0026] Based on the above, embodiments of the present invention provide a method for training a fatty liver grading model, a fatty liver grading method, an electronic device, and a storage medium, which can improve the model's classification accuracy and generalization ability on fuzzy decision boundaries such as mild and moderate fatty liver.

[0027] To implement the process steps and functions of this invention, please refer to [link / reference]. Figure 1 , Figure 1 This is a schematic structural block diagram of an electronic device 100 provided in an embodiment of the present invention. The electronic device 100 includes a memory 101 and a processor 103, which are electrically connected directly or indirectly to each other to realize data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses 104 or signal lines. The memory 101 can be used to store software programs and modules, and the processor 103 executes the software programs and modules stored in the memory 101, thereby performing various functional applications and data processing.

[0028] Electronic device 100 can be, but is not limited to, a personal computer (PC), a server, a distributed computer, etc. It is understood that electronic device 100 is not limited to a physical server, but can also be a virtual machine on a physical server, a virtual machine built on a cloud platform, or any other computer that can provide the same functionality as the server or virtual machine. The operating system of electronic device 100 can be, but is not limited to, Windows, Linux, etc.

[0029] The memory 101 may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0030] The communication connection between the electronic device 100 and external devices is achieved through at least one communication interface 102 (which can be wired or wireless).

[0031] Processor 103 may be an integrated circuit chip with signal processing capabilities. In implementation, the steps of this embodiment can be completed by integrated logic circuits in the hardware of processor 103 or by instructions in software form. Processor 103 may be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0032] Understandable. Figure 1 The structure shown is for illustrative purposes only; the electronic device 100 may also include components that are more advanced than those shown. Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown. Figure 1 The components shown can be implemented using hardware, software, or a combination thereof.

[0033] The following is an exemplary description of the training method for the fatty liver grading model provided in this embodiment of the invention. See also... Figure 2 The subject executing this method can be one of the above. Figure 1 The electronic device 100 shown, the method includes as follows Figure 2 The following steps are described: S21: Obtain the training sample set; the training sample set includes multiple case image packages, each containing multiple ultrasound images of the same case, text descriptions, five-category labels for classifying diffuse lesions, two-category labels for classifying whether it is fatty liver, and four-category labels for classifying the severity of fatty liver.

[0034] Among them, multiple ultrasound images from the same case: multiple ultrasound images from a single ultrasound examination of the case constitute a variable-length image package.

[0035] Text description: The pathological semantic description text corresponding to a single ultrasound examination in this case.

[0036] Five-category label: used to characterize the major category of diffuse lesions to which the case belongs, which can include five categories: normal, fatty liver, cirrhosis, polycystic liver, and hepatomegaly.

[0037] The five-category task uses a mutually exclusive single-label approach. For cases with multiple liver ultrasound phenotypes, a unique five-category label is obtained according to a pre-defined primary label determination rule; the primary label determination rule can be based on the main conclusion of the ultrasound report.

[0038] Binary label: used to characterize whether the case is fatty liver, including two categories: fatty liver and non-fatty liver.

[0039] Four-category labels: used to characterize the severity of fatty liver, including four categories: non-fatty liver, mild fatty liver, moderate fatty liver, and severe fatty liver.

[0040] The "non-fatty liver" category in both the binary and quadrilateral labels uses the same case definition: cases with mild, moderate, or severe fatty liver characteristics are classified as fatty liver, while cases without fatty liver characteristics are classified as non-fatty liver.

[0041] S22: Input the ultrasound image and the corresponding text description into the basic image encoder and the basic text encoder respectively for comparative learning to obtain the initially trained image encoder.

[0042] A dual-tower network consisting of an image encoder and a text encoder is constructed. A batch of ultrasound images is input into the base image encoder to obtain the corresponding image feature vectors; simultaneously, the text descriptions paired with each image in the batch are input into the base text encoder to obtain the corresponding text feature vectors. A symmetric contrastive loss function is used to shorten the distance between paired image-text pairs in the latent feature space, while widening the distance between unpaired image-text pairs. After training, a pre-trained image encoder is obtained, which has the ability to internalize medical semantic priors into visual features.

[0043] S23: Input each case image package into the pre-trained image encoder, extract the feature vectors of each ultrasound image, and stack all feature vectors in the same case image package according to the sequence dimension to obtain a variable-length feature sequence tensor.

[0044] For a case image package containing n ultrasound images, n feature vectors are obtained after the above extraction. All feature vectors within the same case image package are stacked according to their sequence dimension to form a variable-length feature sequence tensor, represented as: X = [x1, x2, ..., x...]. n ], where n is the number of ultrasound images in the case image package, x i Let n be the feature vector of the i-th ultrasound image. The sequence length n corresponding to each case image package can be the same or different.

[0045] S24: Based on the variable-length feature sequence tensor and the corresponding five-class classification labels, supervise the training of the preset basic multi-instance learning network to obtain the liver five-class classification model.

[0046] The variable-length feature sequence tensor corresponding to each case image package is input into a pre-defined basic multi-instance learning network. This network uses an internal attention mechanism to weight and aggregate the features of each instance, outputting the predicted probability distribution of the case image package across five diffuse lesion categories. Using the five-class labels as supervision signals, a classification loss function is calculated, and the network parameters are updated through backpropagation.

[0047] Optionally, the basic multi-instance learning network is a multi-instance aggregation network (TransMIL) based on the Transformer architecture, which can use a self-attention mechanism to model the spatial and logical correlations between various ultrasound sections and output packet-level prediction results through attention pooling.

[0048] After training in this step, the liver five-category model has established a global understanding of the major categories of diffuse lesions (normal, fatty liver, cirrhosis, polycystic liver, and hepatomegaly).

[0049] S25: Replace the classification head of the five-class liver classification model with a two-dimensional classification head, and use the two-class labels as the supervision signal for supervised training to obtain a two-class classification model for fatty liver.

[0050] Inheriting all network weights from the liver five-class classification model trained in step S24, only the classification head at the end is replaced with a two-dimensional output instead of a five-dimensional output. The two-dimensional classification head is used to distinguish between fatty liver and non-fatty liver.

[0051] The labels of the training set were reconstructed: all cases with fatty liver characteristics (including mild, moderate and severe fatty liver) were uniformly classified as positive (label 1), and cases without fatty liver characteristics, including normal, cirrhosis, polycystic liver, hepatomegaly, etc., were uniformly classified as negative (label 0).

[0052] The reconstructed binary classification labels are used as supervision signals to fine-tune the entire network after replacing the classification head. This step reduces the task to binary classification, forcing the network's internal attention mechanism to abandon attention to irrelevant pathological features such as cirrhotic nodules and cysts, and focus the feature response on the visual representation unique to fatty liver.

[0053] S26: Replace the two-dimensional classifier head of the fatty liver binary classification model with a four-dimensional classifier head, and use the four-class labels as supervision signals for supervised training to obtain a fatty liver grading model.

[0054] Inherit all network weights from the fatty liver binary classification model trained in step S25, and replace its two-dimensional classification head with a four-dimensional classification head. The four output dimensions of the four-dimensional classification head correspond to non-fatty liver, mild fatty liver, moderate fatty liver, and severe fatty liver, respectively.

[0055] Using four-class labels as supervision signals, the entire network after replacing the classification head is fine-tuned. Since the binary classification weights in the previous stage have provided the network with precise parameter initialization points, the network already knows exactly what features to look for (i.e., fatty liver characteristics). The gradient update in this stage can focus entirely on calculating the "quantization scale" of these features (i.e., the specific degree of decay), thus smoothly crossing the fuzzy decision boundary between mild and moderate fatty liver.

[0056] Thus, a final fatty liver grading model that can be used for fine-grained grading of fatty liver severity was obtained.

[0057] The training method for this fatty liver grading model, by constructing a training sample set containing three granularity labels, and sequentially executing a progressive training process of contrastive learning pre-training, five-class global cognition establishment, binary classification feature focusing, and four-class fine-grained fine-tuning, has the following beneficial effects: Step S24, the five-class classification training, enables the model to prioritize the establishment of a global understanding of diffuse lesions. Step S25, the two-class classification training, forces the attention mechanism to focus on the specific characteristics of fatty liver. Step S26, the four-class classification training, refines the quantification based on the foundation laid in the first two steps. The three steps form a learning path from coarse to fine, avoiding the gradient confusion problem in the traditional one-step fine-grained classification.

[0058] The cross-modal contrastive learning in step S22 injects medical text semantic priors into the image encoder, giving the attention mechanism of the subsequent multi-instance learning network a clear "semantic compass" and effectively suppressing the diffusion of attention.

[0059] The text length of real clinical ultrasound reports generally exceeds the input truncation threshold of the multimodal encoder (e.g., 77 tokens), and is rich in measurement values ​​unrelated to diffusion grading, descriptions of non-liver organs, and colloquial redundant expressions. Direct input will result in the core pathological semantics being truncated or submerged in characteristic noise.

[0060] For this reason, see Figure 3 In step S21 of this embodiment of the invention, the step of obtaining the text description may include: S211: Obtain the original unstructured electronic medical record text corresponding to each case image package, parse the electronic medical record text, and extract image visual feature descriptions and diagnostic opinions.

[0061] First, obtain the original unstructured electronic medical record texts from the training sample set.

[0062] The system uses a pre-defined natural language processing script to perform field-level parsing on the raw unstructured electronic medical record text. Based on regular expression matching and a keyword dictionary, the script automatically identifies and extracts two core key fields from the medical record text: (1) "Ultrasound findings" field: This is the objective description made by the doctor of the visual features presented on the ultrasound image when reviewing the image. This field contains a linguistic mapping of pixel-level features such as liver morphology, parenchymal echo intensity, liver-kidney contrast, posterior acoustic attenuation, and clarity of ductal structures.

[0063] (2) "Ultrasound Conclusion" field: This is the final diagnostic opinion given by the doctor based on "Ultrasound findings", such as "mild fatty liver", "diffuse liver disease", etc.

[0064] If the above fields in the original medical record text are named differently due to differences in hospital templates (such as "examination description" and "diagnosis result"), they can be matched and mapped adaptively based on the semantic similarity of the fields to ensure the accurate extraction of key information.

[0065] S212: Input the image visual feature description and diagnostic opinions into a pre-trained large language model, perform cross-modal semantic condensation, and obtain text descriptions.

[0066] The “ultrasound findings” and “ultrasound conclusions” extracted in step S211 are used as input sequences and input together into the pre-deployed large language model.

[0067] A dedicated prompt engineering approach for "ultrasound-induced diffuse liver disease" is pre-designed for the large language model. The prompts include clear role settings, task instructions, and output format constraints.

[0068] In an optional implementation, the large language model is instructed to perform the following processing logic: Remove irrelevant organs: Automatically remove descriptions of non-liver-related organs from the report, including but not limited to content related to organs such as the pancreas, spleen, and prostate.

[0069] Filter out focal lesion descriptions: Automatically remove descriptions of space-occupying lesions, including but not limited to focal lesions such as anechoic cysts and hemangiomas.

[0070] Filter out numerical values ​​and template noise: Automatically remove organ size measurement values ​​(such as "liver oblique diameter 13cm"), equipment parameters, patient basic information and routine template phrases.

[0071] Core Feature Recombination: Forcefully extract visual representation words highly related to diffuse liver disease, including but not limited to the echo intensity and distribution characteristics of liver parenchyma, echo contrast between liver and kidney, posterior sound attenuation, and display of intrahepatic tubular structures; recombin the extracted feature words into a standardized pathological description text with high information density.

[0072] Length constraint: The total length of the standardized pathological description text is constrained to not exceed a preset token threshold (e.g., ≤50 tokens) to ensure that it can be fully processed by the basic text encoder in the subsequent step S22 and to avoid truncation.

[0073] After the above steps S211 and S212, the original long noisy report is converted into a clean short text description. This text description retains the core pathological semantics that are most valuable for identifying diffuse lesions in the original report, while eliminating all irrelevant noise, and its length meets the input limit of the text encoder.

[0074] In addition, the raw ultrasound images contain non-medical pixels such as the device UI border and Doppler blood flow signals. Furthermore, the numerous cross-sections acquired in a single examination include many non-standard cross-sections that are irrelevant to echo assessment. Directly inputting the aforementioned irrelevant visual information into the image encoder not only pollutes the image feature space but also makes it difficult for the attention mechanism of the multi-instance learning network to focus on key anatomical regions such as the liver parenchyma and renal cortex when faced with massive variable-length image packages, thus exacerbating attention diffusion.

[0075] To reduce the interference of irrelevant visual noise on the image encoder and accelerate the accurate alignment of image features with text descriptions, embodiments of this invention perform specific purification and visual cue processing on the input image. See also Figure 4 The steps for acquiring multiple ultrasound images from a case image package may include: S213: Remove non-medical UI borders and Doppler modal information from the original ultrasound image, and retain the pure grayscale ultrasound scanning area.

[0076] After acquiring the raw ultrasound images, image cleansing and modality stripping are performed first: UI border stripping: Using image edge detection algorithms and preset ROI (region of interest) mask templates, non-medical UI border areas containing equipment parameters, hospital information, patient privacy information, measurement values ​​and equipment logos around the original ultrasound image are identified and cropped, leaving only the ultrasound beam scanning area, that is, the fan-shaped or rectangular grayscale imaging area in the ultrasound image.

[0077] Doppler mode stripping: This step detects whether a color Doppler blood flow signal layer is superimposed on the original ultrasound image. If present, the color pixel layer is separated from the original image and discarded, retaining only the basic grayscale anatomical structure information. This is because the liver parenchyma echo intensity, liver-kidney contrast, and posterior attenuation, which are relied upon for fatty liver grading, are all grayscale ultrasound features. Color Doppler information is a redundant mode, and its presence would interfere with the encoder's learning of grayscale texture.

[0078] After this step, a clean grayscale ultrasound frame image is obtained, in which the pixel space contains only grayscale information of medical anatomical structures, and all non-medical information and irrelevant modalities are removed.

[0079] S214: Using a pre-trained section classification model, select N standard section images for diffuse lesions from the pure grayscale ultrasound scanning area.

[0080] The clean grayscale image obtained in step S213 is input into the pre-trained section classification model to select the standard section image that is most valuable for the diagnosis of diffuse lesions.

[0081] Optionally, the section classification model is a multi-classification model based on a deep convolutional neural network (such as ResNet) or a lightweight ViT architecture. It is pre-trained on an ultrasound image dataset labeled with section types and is able to identify the section category of the input ultrasound image.

[0082] In one optional implementation, the N standard sections include the following 6 standard sections: the oblique section of the right liver through the first hepatic hilum, the sagittal section of the liver and right kidney, the transverse section below the xiphoid process, the oblique section through the second hepatic hilum, the sagittal section through the abdominal aorta, and the oblique section below the right costal margin through the top of the right hepatic diaphragm.

[0083] This step of forced section screening eliminates a large number of sections that are irrelevant to the assessment of diffuse lesions (such as sections used only to measure ascites depth or only to display the gallbladder), ensuring that each frame of the input sequence of the subsequent multi-instance learning network carries high-density diagnostic information, effectively alleviating the problem of attention diffusion.

[0084] S215: Call the pre-trained target detection model to automatically identify the liver parenchyma and renal cortex regions in each standard cross-sectional image, draw the corresponding bounding boxes on the pixel level of the original image, and use the standard cross-sectional images with bounding boxes as ultrasound images in the case image package.

[0085] Each standard cross-sectional image selected in step S214 is input into a pre-trained target detection model. The target detection model can be a YOLO series model, which is pre-trained on an ultrasound image dataset with bounding boxes of the liver and kidney regions.

[0086] The object detection model performs the following operations on each input standard cross-sectional image: Liver parenchyma region detection: Identify and locate the liver parenchyma region in the image, and output the coordinates of the upper left corner and the lower right corner (x1, y1, x2, y2) of the smallest bounding rectangle of the region.

[0087] Renal cortical region detection: Identify and locate the renal cortical region in the image (limited to standard sections that simultaneously include both the liver and kidney, such as the sagittal section), and output the bounding box coordinates of this region. For standard sections that do not include the kidney (such as the oblique section of the right liver through the first hepatic hilum), this step only outputs the bounding box of the liver region.

[0088] After the detection is completed, the target detection model directly draws the bounding box at the original pixel level, that is, it overlays the rectangular frame lines (for example, using green or red highlighted lines with a line width of 2 to 4 pixels) on the original image, and inputs the standard cross-sectional image with the bounding box as the final ultrasound image in the case image package into subsequent steps.

[0089] The drawing of the explicit bounding box serves as a visual cue, providing a strong spatial prior for the image encoder at the input end. When the image encoder processes the image, the presence of the bounding box allows the model to naturally focus its attentional receptive field on the key contrast space between the liver parenchyma and the kidney cortex, rather than aimlessly scanning the entire image.

[0090] The bounding boxes of the liver and kidney cortex regions drawn in step S215 form a precise spatial semantic correspondence with the condensed pathological text such as "liver-kidney echo contrast" output by the large language model: the textual concept of "liver-kidney echo contrast" guides the model "what to learn," while the bounding boxes of the liver and kidney regions drawn by YOLO guide the model "where to look." Both achieve dual alignment of "textual semantic anchors" and "visual spatial anchors" at the input end of cross-modal contrastive learning, improving the convergence speed and alignment accuracy of image-text contrastive learning.

[0091] In one alternative implementation, see [link to implementation details]. Figure 5 Step S22 may include the following sub-steps: S221: Input a batch of ultrasound images and corresponding text descriptions into the basic image encoder and basic text encoder respectively to obtain the corresponding number of image feature vectors and text feature vectors.

[0092] Obtain a batch from the training sample set, which contains ultrasound images from B case image packages and their corresponding condensed text descriptions, denoted as . .

[0093] Input B ultrasound images into the base image encoder to obtain B corresponding image feature vectors. Simultaneously, B text descriptions are input into the basic text encoder to obtain the corresponding B text feature vectors. .

[0094] The basic image encoder can adopt the ViT (Vision Transformer) or ResNet architecture, while the basic text encoder is based on the Transformer architecture.

[0095] In one optional implementation, both the image feature vector and the text feature vector have a dimension of d (e.g., d=256), and their feature space dimensions are unified after passing through the projection layer.

[0096] S222: After normalizing the image feature vectors and text feature vectors, calculate the cosine similarity between all image feature vectors and text feature vectors within the batch.

[0097] The B image feature vectors obtained in step S221 and B text feature vectors Perform L2 normalization processing separately.

[0098] Calculate the pairwise cosine similarity between all image features and all text features within a batch. .

[0099] S223: Based on cosine similarity, calculate the contrast loss between image and text and the contrast loss between text and image, and use the average of the two as the overall contrast loss for this batch.

[0100] To maximize the similarity of positive sample pairs (i.e. paired images and text) and minimize the similarity of negative sample pairs (i.e. images and text that do not match within a batch), this embodiment of the invention constructs a symmetric InfoNCE contrastive loss function.

[0101] Among them, the contrast loss for image-to-text conversion. Defined as:

[0102] Contrast loss for text-to-image Defined as:

[0103] in, It is a learnable temperature hyperparameter used to adjust the penalty level of the model for difficult negative samples.

[0104] Ultimately, the overall comparative loss for this batch The average of the two:

[0105] S224: Based on the overall contrast loss, update the network weights of the basic image encoder and the basic text encoder through the backpropagation algorithm to obtain the initially trained image encoder.

[0106] Minimize using the backpropagation algorithm The network weights of the image encoder and text encoder are updated synchronously.

[0107] This invention, through the introduction of a formulaic symmetric contrast loss, forces the image encoder not only to learn macroscopic organ contours but also to capture extremely microscopic ultrasound texture features such as "dense dots" and "coarsened echoes," and achieves precise alignment of visual pixels with complex medical semantics in a high-dimensional feature space.

[0108] After completing the variable-length feature sequence extraction and stacking in step S23, this embodiment of the invention uses the variable-length feature sequence tensor and the corresponding five-class classification labels to perform supervised training on a preset basic multi-instance learning network to obtain a liver five-class classification model. This step, through a differentiated parameter update strategy, ensures that the internalized medical semantic priors from the image encoder are preserved during training, while also enabling the multi-instance learning network to learn to focus on the key features of diffuse lesions. See also Figure 6 Step S24 is achieved through the following sub-steps S241 to S244: S241: Input the variable-length feature sequence tensor corresponding to each case image package into the preset basic multi-instance learning network to obtain the classification prediction probability.

[0109] Input the variable-length feature sequence tensors corresponding to each case image package cached in step S23 into the preset basic multi-instance learning network.

[0110] In this embodiment, the basic multi-instance learning network adopts the TransMIL network based on the Transformer architecture. The core structure of this network includes: The Transformer encoder layer performs bidirectional self-attention computation on the input variable-length feature sequence. This encoder layer can model the spatial and logical correlations between ultrasound sections within the same case image package, that is, automatically identify which sections have complementary diagnostic information and which sections have redundant relationships through the self-attention mechanism. Since the feature vectors extracted in step S23 have internalized the semantic prior of medical text through CLIP contrastive learning (i.e., the feature vector of each section carries pathological descriptive information such as "liver parenchymal echo" and "liver-kidney contrast"), TransMIL's attention computation is equivalent to performing feature aggregation under the condition of having a "semantic compass".

[0111] Attention pooling layer: Aggregates the variable-length sequence output by the Transformer encoder into a fixed-length bag-level feature vector. Compared to simple average pooling or max pooling, attention pooling can adaptively assign attention weights to each instance, allowing the model to focus on the most diagnostically valuable aspects within the bag.

[0112] Five-dimensional classification head: Maps package-level features to the predicted probability distribution of five diffuse lesion categories: normal, fatty liver, cirrhosis, polycystic liver, and hepatomegaly.

[0113] After performing the aforementioned forward inference on the input variable-length feature sequence tensor, the TransMIL network outputs the five-class classification prediction probability corresponding to the case image package.

[0114] S242: Freeze all parameters of the image encoder and independently supervise the training of the basic multi-instance learning network based on the five-class labels and classification prediction probabilities until the basic multi-instance learning network initially converges.

[0115] In this stage, all parameters of the initially trained image encoder are completely frozen, preventing it from participating in any gradient updates.

[0116] The five-class classification prediction probabilities obtained in step S241 are compared with the five-class classification labels obtained in step S21, and the classification loss function is calculated. This embodiment of the invention uses standard cross-entropy loss.

[0117] The gradients of each trainable parameter in the basic multi-instance learning network are calculated using the backpropagation algorithm, and the weights of TransMIL are updated using the optimizer. Since all parameters of the image encoder are frozen, the gradient information is not propagated back to the image encoder; it is only used to optimize the attention mechanism and classification head parameters within the TransMIL network.

[0118] The forward inference and backpropagation processes described above are executed iteratively until the TransMIL network initially converges on the five-class classification task. Through independent training in this stage, the TransMIL network learns to model the correlation between each ultrasound section and assign attention weights based on the feature sequence containing CLIP semantic priors.

[0119] S243: Physically stitch the image encoder with the initially converged base multi-instance learning network, and unfreeze the parameters of the last M network layers near the output of the image encoder.

[0120] After the TransMIL network converges independently in step S242, the image encoder is physically spliced ​​with the preliminarily converged base multi-instance learning network to construct a complete end-to-end classification network. "Physical splicing" refers to connecting the output of the image encoder to the input of the base multi-instance learning network, allowing the output features of the image encoder to be directly input into the base multi-instance learning network during subsequent training, and enabling gradients to be backpropagated from the base multi-instance learning network to the unfrozen portion of the image encoder.

[0121] To prevent damage to the universal multimodal alignment feature space established by the image encoder in step S22, this embodiment of the invention does not adopt a full unfreezing approach, but selectively unfreezes only the parameters of the last M (e.g., M=5) network layers near the output of the image encoder, allowing them to participate in gradient updates; the remaining shallow network layers remain frozen.

[0122] The final M layer is thawed so that only the deep layers are fine-tuned for medical image features, while the shallow general features remain unchanged. This achieves both adaptation to downstream tasks and preservation of alignment priors to the greatest extent, effectively avoiding catastrophic forgetting.

[0123] S244: Differential learning rate is used to perform end-to-end joint supervised training with five-class labels as supervision signals to obtain a liver five-class classification model; among them, the learning rate of the basic multi-instance learning network is higher than the learning rate of the unfrozen part of the image encoder.

[0124] A differential learning rate strategy is adopted, using the five-class labels as supervision signals, to jointly supervise the training of the physically spliced ​​end-to-end network.

[0125] Set different learning rates for different parts of the network, for example, set the learning rate for a basic multi-instance learning network. Set the learning rate for the last M layers of the image encoder after it is unfrozen. .

[0126] The technical significance of the aforementioned differential learning rate setting lies in the following: The TransMIL network, as the core aggregation module of the task, needs to quickly adapt to the five-class classification task with a normal learning rate; the image encoder has already obtained good feature representation capabilities through large-scale contrastive learning, and only needs local fine-tuning with a lower learning rate to avoid significant deviation from the already aligned feature space. If the same learning rate is applied to both, the feature space of the image encoder may be excessively distorted by the downstream classification task, causing the cross-modal alignment prior established in step S22 to be destroyed.

[0127] The supervision signal in this stage still adopts the five-class cross-entropy loss in step S242, but the gradient is backpropagated through the TransMIL network and then further propagated back to the last M layers of the image encoder that have been unfrozen in order to fine-tune its parameters.

[0128] The above joint training is performed iteratively until the model converges on the five-class classification task, thus obtaining the liver five-class classification model.

[0129] Through the differentiated training strategy in steps S241 to S244, the obtained liver five-classification model not only retains the medical semantic guidance capability of the image encoder given by CLIP cross-modal contrastive learning, but also enables the attention mechanism of the TransMIL network to establish a preliminary understanding of the major categories of diffuse lesions in the five-classification task, laying a smooth parameter initialization foundation for the subsequent binary classification focusing in step S25 and fine-grained fine-tuning in step S26.

[0130] In one alternative implementation, step S25 may include: S251: Inherit all weights of the liver five-class classification model and replace the classification head of the liver five-class classification model with a two-dimensional classification head; the two-dimensional classification head is used to distinguish between fatty liver and non-fatty liver.

[0131] Load all network weights of the liver five-class classification model after training in step S24, remove the five-dimensional classifier head at the end of the liver five-class classification model, and replace it with a two-dimensional classifier head. The output dimension of the two-dimensional classifier head is 2, corresponding to the two categories of "fatty liver" and "non-fatty liver" respectively. Its output is activated by Softmax to obtain the predicted probabilities of the two categories.

[0132] Because the parameter dimensions of the classification head have changed (from 5-dimensional to 2-dimensional), the parameters of the 2D classification head are randomly initialized or zero-initialized, and do not inherit the weights of the original 5-dimensional classification head.

[0133] S252: The liver five-class classification model after the classification head is replaced is trained with binary classification labels as supervision signals, so that the attention mechanism of the multi-instance learning network in the liver five-class classification model converges to the fatty liver-specific visual representation, thus obtaining a fatty liver binary classification model.

[0134] The binary classification labels are used as supervision signals to supervise the training of the entire model after the classification head is replaced.

[0135] The variable-length feature sequences of each case image package are input into the weighted network. After aggregation by the TransMIL network, the two-dimensional classification head outputs the binary classification prediction probabilities of "fatty liver" and "non-fatty liver". The binary classification labels are used as supervision signals to calculate the binary classification cross-entropy loss.

[0136] Gradients are calculated and network weights are updated via backpropagation. During training in this step, the last M layers of the image encoder use the same low learning rate as in step S244 (e.g., 1×10⁻⁶). -5 The TransMIL network uses a high learning rate (e.g., 1×10⁻⁶).-4 ), continuing the differential learning rate strategy.

[0137] As the binary classification training iterations proceed, the model's attention mechanism undergoes the following changes: Since the optimization objective of the binary classification task only requires distinguishing between "presence or absence of fatty liver" and not between other diffuse lesions such as cirrhosis and polycystic liver, the self-attention weights and aggregate weights of the attention pooling layer within the TransMIL network, driven by the loss function, will gradually discard responses to non-fatty liver lesion features such as cirrhotic nodules, liver cysts, and polycystic kidney disease, compressing the instance weights corresponding to these irrelevant features to near zero. Simultaneously, the model will force attention resources to focus on the visual representations specific to fatty liver, including: Liver and kidney echo contrast: the degree of enhancement of liver parenchyma echo relative to renal cortex echo; Posterior acoustic attenuation: The degree of attenuation of the far-field echo after the ultrasound beam passes through the liver; Clarity of intrahepatic duct structures: visualization of the walls of the portal vein and hepatic veins.

[0138] The above three types of features constitute the core visual basis for grading fatty liver. When the attention weights of the TransMIL network are fully focused on the ultrasound sections and pixel regions corresponding to the above three types of features, the feature response distribution within the model completes the transfer from "global cognition of diffuse lesions" to "specific cognition of fatty liver", at which point the binary classification task converges.

[0139] This step occupies a crucial, bridging position in the entire three-stage course: Building upon the previous step: the global understanding of diffuse lesions established in the five-category stage of step S24 does not require learning basic characteristics such as "what is the liver" from scratch.

[0140] The next step is to gradually focus the model's attention on the specific characteristics of fatty liver through a binary classification task. Before entering the most difficult fine-grained four-class classification stage, the model has already completed the "dimensionality reduction and focusing" of the feature space in the parameter space. The model clearly knows "where to look and what features to find". The gradient updates in the subsequent four-class classification stage will be entirely dedicated to quantifying the "degree scale" of these features, completely eliminating the gradient confusion caused by "simultaneously finding features and classifying degrees" when directly performing four-class classification.

[0141] In real-world clinical data, the sample size of fatty liver with different severity levels exhibits a natural long-tail distribution: mild fatty liver is the most common, followed by moderate, while severe fatty liver is relatively rare due to its lower incidence. If a conventional cross-entropy loss function is used, the model will naturally favor the larger sample size of mild fatty liver during training, neglecting its ability to distinguish severe samples. Furthermore, because the decision boundaries between adjacent levels are blurred, the cross-entropy loss lacks additional optimization guidance for easily confused samples located near these boundaries (e.g., the boundary between mild and moderate), further exacerbating the decline in classification accuracy.

[0142] Therefore, in one alternative implementation, step S26 may include: S261: Inherit all weights of the fatty liver binary classification model and replace the two-dimensional classification head of the fatty liver binary classification model with a four-dimensional classification head; the four dimensions of the four-dimensional classification head correspond to non-fatty liver, mild fatty liver, moderate fatty liver and severe fatty liver, respectively.

[0143] Load all network weights from the fatty liver binary classification model trained in step S25, remove the two-dimensional classifier head at the end of the model, and replace it with a four-dimensional classifier head. The output dimension of the four-dimensional classifier head is 4, corresponding to: non-fatty liver, mild fatty liver, moderate fatty liver, and severe fatty liver. After Softmax activation, its output yields the predicted probabilities for the four categories, corresponding to the four categories mentioned above.

[0144] The parameters of the four-dimensional classification head are initialized randomly or with zero initialization, and do not inherit the weights of the original binary classification head.

[0145] S262: Using four-class labels as supervision signals and the focus loss function as the optimization objective, supervised training is performed on the fatty liver binary classification model after replacing the classification head to obtain a fatty liver grading model.

[0146] In the focus loss function, the weight of severe fatty liver is greater than that of moderate fatty liver, greater than that of mild fatty liver, and greater than that of non-fatty liver.

[0147] Using four-class labels as supervision signals and the focus loss function as the optimization objective, the entire network after the classifier head is replaced is trained under supervision.

[0148] The variable-length feature sequences of each case image package are processed by an image encoder to extract features, and then fed into a TransMIL network with inherited weights for aggregation. The four-dimensional classification head outputs the predicted probabilities of four categories. The four-category labels are used as supervision signals to calculate the focal loss.

[0149] The focus loss function is defined as:

[0150] in, For the model to the true category The predicted probability, The focusing parameter is used to adjust the model's attention to easily classified and difficult-to-classify samples. This is the class weighting factor, used to adjust the contribution ratio of different class samples to the total loss.

[0151] To address the naturally uneven long-tail distribution of fatty liver samples of different severity levels in this invention, this invention sets a category weighting factor. The following constraints must be satisfied:

[0152] That is, the severe fatty liver category has the highest weight, followed by moderate, then mild, and the non-fatty liver category has the lowest weight.

[0153] Finally, the gradient is calculated and the network weights are updated through backpropagation. This process is repeated iteratively until the four-class focal loss converges, thus obtaining the final fatty liver grading model.

[0154] This concludes the three-stage course consisting of S24 (five-category classification), S25 (two-category classification), and S26 (four-category classification). The internal logic of the three stages can be summarized as follows: S24 Five-class classification: Establish a broad-domain representation of liver ultrasound phenotypes, enabling the model to learn the general characteristics of different liver ultrasound phenotypes.

[0155] S25 binary classification: guides attention to fatty liver-specific features, allowing the model to learn to "focus only on fatty liver features".

[0156] S26 Four-Class Classification: Based on focus, it refines quantification, enabling the model to learn to "distinguish the severity of fatty liver".

[0157] If S25 is skipped and the process proceeds directly from S24 to S26, the model needs to optimize simultaneously across two challenging tasks: "feature finding" and "gradation." This results in an adversarial relationship between gradient directions, leading to gradient confusion. However, the three-stage process of this invention ensures that each stage has a single, clear optimization objective. The convergence result of the previous stage provides precise parameter initialization points for the next stage, greatly smoothing the loss function space and enabling the model to ultimately complete the fine-grained grading task with high accuracy.

[0158] To verify the technical effectiveness of the progressive learning process consisting of five-class pre-training, two-class transitional training, and four-class target training adopted in the embodiments of the present invention, the experimental results are as follows, under the same settings of data partitioning, image encoder, packet-level aggregation network, optimizer, number of training rounds, and evaluation criteria: The experimental dataset is shown in Table 1: Table 1

[0159] The data above comes from 327 data source centers. Each data subset is divided according to the patient dimension, and the training, validation, and test sets are divided in an 8:1:1 ratio. All ultrasound images of the same patient are only included in one data subset of the training, validation, or test set.

[0160] The core comparative experiment of the course learning process: Groups A through D were set up. Group A skipped the five-class and two-class classification stages and directly trained using four-class labels; Group B skipped the two-class classification transition stage and directly entered the four-class classification stage; Group C skipped the five-class classification stage and only went through the two-class classification stage; Group D adopted the complete "five-class classification → two-class classification → four-class classification" learning process of this invention. All groups used the same data augmentation, loss calculation methods, and test set. The experimental results are shown in Table 2. Table 2

[0161] Wherein, ACC represents the overall classification accuracy; P, R, and F1 represent the macro-average precision, macro-average recall, and macro-average F1 score of the four-class classification task, respectively. P, R, and F1 are all expressed in decimal form from 0 to 1, and the same calculation method is used for all experimental groups.

[0162] As shown in Table 2, the ACC, F1, P, and R of group D, which adopted the complete course learning process, were 89.76%, 0.8537, 0.8446, and 0.8674, respectively. Compared with group A, which directly implemented four-class classification, group D's ACC improved by 2.93 percentage points and its F1 improved by 3.07 percentage points; compared with group B, which skipped the binary classification transition stage, group D's ACC improved by 0.78 percentage points and its F1 improved by 1.63 percentage points. These results support the ability of binary classification transition training to further focus on fatty liver-related features based on the broad representation established in the five-class classification stage, and to provide relatively stable parameter initialization for subsequent fine-grained four-class classification training, thereby improving the overall classification performance of the model.

[0163] Furthermore, embodiments of the present invention also provide a method for grading fatty liver, see [link to relevant documentation]. Figure 7 The grading method for fatty liver includes: S31: Obtain the image package of cases to be graded; the image package of cases to be graded includes multiple ultrasound images of the same case.

[0164] S32: Input the image package of cases to be graded into the fatty liver grading model to obtain the grading results of the image package of cases to be graded; the fatty liver grading model is trained by the above-mentioned fatty liver grading model training method.

[0165] In summary, the fatty liver grading model training method, fatty liver grading method, electronic device, and storage medium provided by this invention introduce a large language model to perform cross-modal semantic condensation on the original unstructured ultrasound report, compressing lengthy text into a high-information-density pathological description with controllable length. This avoids the loss of core semantics caused by input truncation in conventional multimodal encoders. Furthermore, by combining UI borders, Doppler modality stripping, standard section filtering, and YOLO object detection boxes on the image end as explicit visual cues, a precise correspondence between "textual semantic anchors" and "visual spatial anchors" is formed at the input end, significantly improving the convergence speed and alignment accuracy of image-text comparison learning. This replaces the image encoder with medical semantic priors... Traditional pure visual encoders, combined with a differential parameter update strategy of frozen independent training and differential learning rate, give the TransMIL network a clear "semantic compass" when focusing attention within a bag, effectively suppressing attention diffusion and preventing the model from being interfered with by ultrasound artifacts or non-target focal lesions. Through a three-stage progressive learning process of "five-class, two-class, and four-class," the model follows the clinical cognitive pattern of "first establishing a global understanding of diffuse lesions, then forcibly focusing on the specific characteristics of fatty liver, and finally finely quantifying the severity scale" to complete training. This ensures that the optimization objective of each stage is singular and clear, and the convergence result of the previous stage provides a smooth and accurate parameter initialization point for the next stage, effectively alleviating the gradient confusion problem.

[0166] In the embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative; for example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0167] In addition, the functional modules in the various embodiments of the present invention can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0168] If the functionality is implemented as a software module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a computer-readable storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0169] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0170] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A method for training a fatty liver grading model, characterized in that, include: Obtain a training sample set; the training sample set includes multiple case image packages, each containing multiple ultrasound images of the same case, text descriptions, a five-category label for classifying diffuse lesions, a two-category label for classifying whether it is fatty liver, and a four-category label for classifying the severity of fatty liver. The ultrasound image and its corresponding text description are respectively input into the basic image encoder and the basic text encoder for comparative learning to obtain a preliminarily trained image encoder. Each case image package is input into the pre-trained image encoder to extract the feature vectors of each ultrasound image, and all feature vectors in the same case image package are stacked according to the sequence dimension to obtain a variable-length feature sequence tensor. Based on the variable-length feature sequence tensor and the corresponding five-class label, a pre-defined basic multi-instance learning network is trained under supervision to obtain a liver five-class classification model; the liver five-class classification model includes a five-dimensional classification head, which is used to output the predicted probabilities of five diffuse lesion categories; Inherit all network weights of the liver five-classification model except for the five-dimensional classification head, replace the five-dimensional classification head with a two-dimensional classification head, and perform supervised training with the two-dimensional classification label as the supervision signal to obtain a fatty liver two-classification model; the two-dimensional classification head is used to distinguish between fatty liver and non-fatty liver, and the parameters of the two-dimensional classification head are randomly initialized or zero-initialized; Inheriting all network weights of the fatty liver binary classification model except for the two-dimensional classifier, the two-dimensional classifier is replaced with a four-dimensional classifier. Supervised training is performed using the four-dimensional classification labels as supervision signals to obtain a fatty liver grading model. The four-dimensional classifier is used to output the predicted probabilities of four categories: non-fatty liver, mild fatty liver, moderate fatty liver, and severe fatty liver, respectively. The parameters of the four-dimensional classifier are randomly initialized or zero-initialized.

2. The method according to claim 1, characterized in that, The steps for obtaining the text description include: Obtain the original unstructured electronic medical record text corresponding to each case image package, parse the electronic medical record text, and extract image visual feature descriptions and diagnostic opinions; The image visual feature description and the diagnostic opinion are input into a pre-trained large language model to perform cross-modal semantic condensation and obtain the text description.

3. The method according to claim 1, characterized in that, The steps for acquiring multiple ultrasound images from the case image package include: Remove non-medical UI borders and Doppler modal information from the original ultrasound image, and retain the pure grayscale ultrasound scanning area; Using a pre-trained section classification model, N standard section images for diffuse lesions are selected from the pure grayscale ultrasound scanning area. The pre-trained target detection model is invoked to automatically identify the liver parenchyma and renal cortex regions in each of the standard cross-sectional images, and corresponding bounding boxes are drawn on the original image pixel level. The standard cross-sectional images with bounding boxes are used as ultrasound images in the case image package.

4. The method according to claim 1, characterized in that, The step of inputting the ultrasound image and its corresponding text description into a base image encoder and a base text encoder respectively for comparative learning to obtain a preliminarily trained image encoder includes: A batch of ultrasound images and their corresponding text descriptions are input into the basic image encoder and the basic text encoder, respectively, to obtain the corresponding number of image feature vectors and text feature vectors; After normalizing the image feature vector and the text feature vector, the cosine similarity between all image feature vectors and text feature vectors within the batch is calculated. Based on the cosine similarity, the image-to-text contrast loss and the text-to-image contrast loss are calculated, and the average of the two is used as the overall contrast loss for this batch. Based on the overall contrast loss, the network weights of the basic image encoder and the basic text encoder are updated using the backpropagation algorithm to obtain the initially trained image encoder.

5. The method according to claim 1, characterized in that, The step of supervising the training of a pre-defined basic multi-instance learning network based on the variable-length feature sequence tensor and the corresponding five-class classification labels to obtain a liver five-class classification model includes: The variable-length feature sequence tensor corresponding to each case image package is input into a preset basic multi-instance learning network to obtain the classification prediction probability. Freeze all parameters of the image encoder, and independently supervise the training of the basic multi-instance learning network based on the five-class labels and the classification prediction probabilities until the basic multi-instance learning network initially converges. The image encoder is physically spliced ​​with the initially converged basic multi-instance learning network, and the parameters of the last M network layers near the output of the image encoder are unfrozen. A liver five-class classification model is obtained by end-to-end joint supervision training using differential learning rate and the five-class classification labels as supervision signals; wherein the learning rate of the basic multi-instance learning network is higher than the learning rate of the unfrozen part of the image encoder.

6. The method according to claim 1, characterized in that, The step of using the binary classification labels as supervisory signals to perform supervised training to obtain a fatty liver binary classification model includes: The liver five-class classification model after the classification head is replaced is trained using the binary classification labels as supervision signals. This causes the attention mechanism of the multi-instance learning network in the liver five-class classification model to converge to the fatty liver-specific visual representation, thus obtaining a fatty liver binary classification model.

7. The method according to claim 1, characterized in that, The process of obtaining a fatty liver grading model by using the four-category labels as supervisory signals for supervised training includes: Using the four-class labels as supervision signals and the focus loss function as the optimization objective, the fatty liver binary classification model after replacing the classification head is trained under supervision to obtain a fatty liver grading model. In the focus loss function, the weight of severe fatty liver is greater than that of moderate fatty liver, the weight of moderate fatty liver is greater than that of mild fatty liver, and the weight of mild fatty liver is greater than that of non-fatty liver.

8. A method for grading fatty liver, characterized in that, include: Obtain the image package of cases to be graded; The image package of cases to be graded includes multiple ultrasound images of the same case; The image package of cases to be graded is input into the fatty liver grading model to obtain the grading result of the image package of cases to be graded; the fatty liver grading model is trained by the fatty liver grading model training method as described in any one of claims 1 to 7.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the method according to any one of claims 1 to 7.

10. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1 to 7.

Citation Information

Patent Citations

  • Fundus image feature processing and analyzing method based on twin network architecture

    CN111833334A

  • Apparatus and method for classifying domain non-specific images using text

    US20240346812A1