A liver focal lesion classification method and system based on ultrasonic contrast

Abstract

This application discloses a method and system for classifying focal liver lesions based on contrast-enhanced ultrasound (CEUS), relating to the fields of artificial intelligence and intelligent diagnostic technology. The method includes: acquiring data to be processed; extracting keyframes from a CEUS dynamic video, performing optical flow processing on the extracted keyframes to form an optical flow feature map; inputting the optical flow feature map and annotated single-frame CEUS images into a dual-flow model to obtain the disease probabilities of five types of diseases; inputting the optical flow feature map into a preset binary sub-model to obtain the positive probabilities of four positive markers; performing correlation coefficient filtering based on the clinical structured data to obtain a clinical feature vector; and performing feature fusion on the probabilities of the five types of diseases, the positive probabilities of the four positive markers, and the clinical feature vector, followed by OT determination to obtain the classification results of focal liver lesions. The classification results of focal liver lesions include: primary hepatocellular carcinoma, liver metastases, intrahepatic cholangiocarcinoma, HH, HA, and OT.

Description

Technical Field

[0001] This specification relates to the fields of artificial intelligence and intelligent diagnostic technology, and in particular to a method and system for classifying focal liver lesions based on ultrasound contrast imaging. Background Technology

[0002] Focal liver lesions (FLL) are a very common type of liver abnormality in clinical practice, encompassing benign, malignant, and borderline lesions, including primary hepatocellular carcinoma, liver metastases, intrahepatic cholangiocarcinoma, hemangioma, liver abscess, and other types. Different types of FLL exhibit significant differences in pathological characteristics, biological behavior, treatment strategies, and prognosis; therefore, accurate multiclassification of FLL is a crucial prerequisite for rational diagnosis and treatment.

[0003] In existing technologies, contrast-enhanced ultrasound (CEUS) offers significant advantages for FLL diagnosis, including being non-invasive, real-time, safe, economical, and providing high spatiotemporal resolution. However, this method is highly dependent on physician experience, has limited ability to identify atypical and rare lesions, and suffers from poor interpretation consistency among different doctors and centers. Furthermore, the large amount of dynamic video information, coupled with low efficiency of manual analysis and susceptibility to fatigue-related misinterpretations, makes it difficult to meet the needs of large-scale screening. Simultaneously, significant differences in equipment models and imaging parameters lead to inconsistent imaging manifestations of the same lesion, insufficient cross-center stability, and difficulty in achieving accurate and standardized multi-classification diagnosis.

[0004] Therefore, there is a need for a classification method and system for focal liver lesions based on ultrasound contrast imaging. Summary of the Invention

[0005] This specification provides a method and system for classifying focal liver lesions based on ultrasound contrast imaging, which is used to solve the above-mentioned technical problems.

[0006] To solve the above-mentioned technical problems, the embodiments in this specification are implemented as follows: This specification provides an embodiment of a method for classifying focal liver lesions based on contrast-enhanced ultrasound, including: Acquire the data to be processed, which includes: CEUS dynamic video and corresponding clinical structured data; Key frames of the CEUS dynamic video are extracted, and optical flow processing is performed on the extracted key frames to form an optical flow feature map. The optical flow feature map and the labeled single-frame CEUS image are input into the dual-flow model to obtain the disease probabilities of five types of diseases, including: the probability of primary hepatocellular carcinoma, the probability of liver metastasis, the probability of intrahepatic cholangiocarcinoma, the probability of hepatic hemangioma, and the probability of liver abscess. The optical flow feature map is input into a preset binary sub-model to obtain the positive probabilities of four positive markers, which include: Hep, GPC-3, CK7, and CK19. Based on the aforementioned structured clinical data, correlation coefficients are used for screening to obtain clinical feature vectors; After fusing the probabilities of the five types of diseases, the positive probabilities of the four positive markers, and the clinical feature vectors, OT determination is performed to obtain the classification results of focal liver lesions.

[0007] This specification also provides an embodiment of a liver focal lesion classification system based on ultrasound contrast imaging, including: The data acquisition module acquires the data to be processed, which includes: CEUS dynamic video and corresponding clinical structured data; The video preprocessing module extracts key frames from the CEUS dynamic video, performs optical flow processing on the extracted key frames, and forms an optical flow feature map. The disease module inputs the optical flow feature map and the labeled single-frame CEUS image into the dual-flow model to obtain the disease probabilities of five types of diseases, including: the probability of primary hepatocellular carcinoma, the probability of liver metastasis, the probability of intrahepatic cholangiocarcinoma, the probability of hepatic hemangioma, and the probability of liver abscess. The pathology module inputs the optical flow feature map into a preset binary sub-model to obtain the positive probabilities of four positive markers, which include: Hep, GPC-3, CK7, and CK19. The clinical module performs correlation coefficient filtering based on the aforementioned structured clinical data to obtain clinical feature vectors; The feature fusion and classification module fuses the probabilities of the five types of diseases, the positive probabilities of the four positive markers, and the clinical feature vectors, and then performs OT determination to obtain the classification results of focal liver lesions.

[0008] The liver focal lesion classification method based on contrast-enhanced ultrasound (CEUS) provided in this specification involves acquiring data to be processed, including: CEUS dynamic video and corresponding clinical structured data; extracting keyframes from the CEUS dynamic video, performing optical flow processing on the extracted keyframes to form an optical flow feature map; inputting the optical flow feature map and annotated single-frame CEUS images into a dual-flow model to obtain the disease probabilities of five categories of diseases, including: the probability of primary hepatocellular carcinoma, the probability of liver metastases, the probability of intrahepatic cholangiocarcinoma, the probability of hepatic hemangioma, and the probability of liver abscess; and inputting the optical flow feature map into a preset binary sub-model to obtain four... The positive probability of four positive biomarkers, including Hep, GPC-3, CK7, and CK19, is used to screen the clinical structured data based on correlation coefficients to obtain a clinical feature vector. The probabilities of the five disease categories, the positive probabilities of the four positive biomarkers, and the clinical feature vector are fused together to determine OT (Organic Lesion) and obtain the classification result of focal liver lesions. This method can accurately diagnose multiple types of FLL (Follicular Ligamentary Liver Disease) based on multi-data fusion, overcoming the drawbacks of relying entirely on physician experience. It also features multi-classification capability, cross-vendor stability, low computational requirements, low cost, and ease of use, possessing enormous potential for large-scale clinical application. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 A schematic diagram of the system architecture for a method of classifying focal liver lesions based on ultrasound contrast imaging, provided in the embodiments of this specification; Figure 2 A flowchart illustrating a method for classifying focal liver lesions based on ultrasound contrast imaging, provided in the embodiments of this specification; Figure 3 This is a schematic diagram of the keyframe extraction process provided in the embodiments of this specification; Figure 4 This is a schematic diagram of a liver focal lesion classification system based on ultrasound contrast imaging, provided as an embodiment of this specification.

[0011] Explanation of reference numerals in the attached figures: The system architecture consists of 100 components, 101 terminal devices, 102 network components, 103 server components, 401 data acquisition module, 403 video preprocessing module, 405 disease module, 407 pathology module, 409 clinical module, and 411 feature fusion and classification module. Detailed Implementation

[0012] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this application.

[0013] Figure 1 This is a schematic diagram of the system architecture for a method of classifying focal liver lesions based on ultrasound contrast imaging, provided as an embodiment of this specification. Figure 1 As shown, system architecture 100 may include at least one terminal device 101, a network 102, and a server 103. Network 102 serves as the medium for providing a communication link between terminal device 101 and server 103. Network 102 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0014] Terminal device 101 interacts with server 103 via network 102 to receive or send messages, etc. Various client applications can be installed on terminal device 101, such as specialized programs for classifying focal liver lesions based on ultrasound contrast imaging.

[0015] Terminal device 101 can be hardware or software. When terminal device 101 is hardware, it can be various dedicated or general-purpose electronic devices, including but not limited to smartphones, tablets, laptops, and desktop computers. When terminal device 101 is software, it can be installed in the electronic devices listed above. It can be implemented as multiple software programs or software modules (e.g., multiple software programs or software modules used to provide distributed services) or as a single software program or software module.

[0016] Server 103 can be a server that provides various services, such as a backend server that provides services to client applications installed on terminal device 101. For example, the server can perform ultrasound-based classification of focal liver lesions so that the results of the ultrasound-based classification of focal liver lesions can be displayed on terminal device 101.

[0017] Server 103 can be either hardware or software. When server 103 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When server 103 is software, it can be implemented as multiple software programs or software modules (e.g., multiple software programs or software modules used to provide distributed services), or as a single software program or software module.

[0018] Figure 2 This is a flowchart illustrating a method for classifying focal liver lesions based on ultrasound contrast imaging, provided in an embodiment of this specification. From a programming perspective, the execution entity of the process can be a program hosted on an application server or application terminal. It is understood that this method can be executed by any device, equipment, platform, or cluster of devices with computing and processing capabilities. Figure 2 As shown, this classification method for focal liver lesions includes: Step S201: Obtain the data to be processed, which includes: CEUS dynamic video and corresponding clinical structured data.

[0019] In this embodiment, the CEUS dynamic video includes CEUS videos of the arterial phase, portal venous phase, and delayed phase. The clinical structured data includes age, sex, tumor size, medical history, serological markers, and pathological information. The medical history includes: history of hepatitis, history of malignant tumors, hypertension, diabetes, smoking, and alcohol consumption. Serological markers include: white blood cell count (WBC), platelet count (PLT), alanine aminotransferase (ALT), aspartate aminotransferase (AST), total protein (TP), albumin (ALB), alpha-fetoprotein (AFP), carcinoembryonic antigen (CEA), cancer antigen-199 (CA-199), and cancer antigen-125 (CA-125). Pathological information includes the expression of biomarkers Hep, GPC-3, CK7, and CK19.

[0020] Hep is a highly specific hepatocellular marker. Most primary hepatocellular carcinomas (HCCs) are Hep-positive, while metastatic liver cancers typically do not express Hep or express very little Hep. Therefore, Hep testing can effectively differentiate hepatocellular tumors from those of other origins and helps distinguish between benign and malignant non-celled liver disease (FLL). However, some benign FLLs (such as hepatocellular adenomas and nodular hyperplasia) may also be Hep-positive. A positive Hep result is mainly characterized by granular or diffuse staining in the cytoplasm, with moderate to strong staining intensity; a negative result is characterized by no obvious staining reaction in the cytoplasm of nodular cells, or extremely weak staining that is difficult to distinguish from the background.

[0021] GPC-3 is a cell surface protein that is often highly expressed in primary hepatocellular carcinoma (HCC), with approximately 70-80% of HCC cases showing a positive expression. Benign flaccid liver disease (FLL) typically does not express GPC-3, and most liver metastases are also GPC-3 negative. A positive GPC-3 result is a brownish staining of the cell membrane and / or cytoplasm, with moderate to strong intensity; a negative result shows no obvious staining reaction in the tumor cell membrane and cytoplasm. CK-7 is an intermediate filament protein, mainly found in various epithelial cells. Its expression helps differentiate different types of liver lesions. Primary HCC is usually CK-7 negative, but partial positive expression is occasionally seen in mixed or poorly differentiated primary HCC; intrahepatic cholangiocarcinoma is usually CK-7 positive; liver metastases from the breast, lung, gastrointestinal tract, etc., may be CK-7 positive; while benign FLL is usually CK-7 negative. CK-19 is a cytoskeletal protein, and its expression is of significant value in the classification of liver lesions. Primary hepatocellular carcinoma is usually CK-19 negative, but some mixed or poorly differentiated primary hepatocellular carcinomas may be CK-19 positive; intrahepatic cholangiocarcinoma is usually CK-19 positive; some liver metastases from the digestive tract may also express CK-19; some benign FLLs containing bile duct components may also be CK-19 positive.

[0022] Step S203: Extract key frames from the CEUS dynamic video, perform optical flow processing on the extracted key frames, and form an optical flow feature map.

[0023] In this embodiment of the specification, the step of extracting key frames from the CEUS dynamic video and performing optical flow processing on the extracted key frames to form an optical flow feature map specifically includes: The CEUS dynamic video is annotated to obtain the ROI video; Based on the time intensity curve, key frames are extracted from the ROI video to obtain the extracted key frames; Based on the differences between two adjacent frames of the extracted keyframes, the optical flow feature map is generated using optical flow technology.

[0024] In the embodiments of this specification, a Siamese network is composed of a template branch and a search branch to extract features from the template and search regions, respectively. Specifically, the template branch is responsible for extracting features of the target object from the labeled image, while the search branch extracts features of potential target regions in the current frame. By calculating the similarity between the features extracted by these two branches, the model can track targets in consecutive frames. Simultaneously, to address the shortcomings of traditional Siamese networks in handling target motion, a temporal motion attention module is introduced. This module utilizes optical flow information to predict the target's displacement and models the target's motion state using a Kalman filter. The Kalman filter can effectively handle noisy motion data, thereby providing more accurate target position predictions.

[0025] In the embodiments of this specification, the step of annotating the CEUS dynamic video to obtain the ROI video specifically includes: The first frame of the CEUS dynamic video is labeled with ROI as a template, and the template branch is input to extract fixed-dimensional lesion template features. The template branch is composed of a CNN backbone network composed of multiple layers of ConvBNReLU modules stacked together. Each frame in the CEUS dynamic video except the first frame is taken as the current frame. The search region image of the current frame is received with the lesion historical location as the center. The search branch is input and the candidate region feature map is extracted. The search branch and the template branch share weights and have the same structure as the CNN backbone network. The template features and the candidate region feature map are cross-correlated and convolved to generate a two-dimensional similarity heatmap. The peak coordinates in the two-dimensional similarity heatmap are used as the initial predicted position of the lesion in the current frame. The initial predicted position of the lesion in the current frame is corrected for the lesion center by time-motion attention, and a rectangular ROI box is generated according to a fixed outer expansion pixel to obtain the lesion ROI image of the current frame. Arrange the lesion ROI images of each current frame in an orderly manner to obtain the ROI video; In a specific embodiment, the fixed outer pixel is selected as 20 pixels, but the specific value can be determined according to the business scenario.

[0026] The process of generating the optical flow feature map using optical flow technology based on the differences between two adjacent frames of the extracted keyframes specifically includes: Based on the two adjacent frames of the extracted keyframe, the pixel-level optical flow field is calculated and the optical flow map is output. The optical flow map includes: horizontal displacement component and vertical displacement component. The optical flow map is corrected by the state equation of a Kalman filter to generate the optical flow feature map.

[0027] In the embodiments described in this specification, the state equation of the Kalman filter is: ; in, This represents the target's state vector at time t; Represents the state transition matrix; This indicates process noise.

[0028] To further understand the generation process of the optical flow feature map provided in this embodiment, it will be explained below with reference to specific embodiments. Specifically, as follows... Figure 3 As shown.

[0029] like Figure 3 As shown, the largest frame of the tumor in the arterial phase is selected to delineate the tumor boundary. A rectangular frame is then automatically generated, and an outward 20-pixel boundary is used to automatically label the CEUS video frame by frame, generating a ROI video. Then, a time-intensity curve (TIC) is plotted using the brightness of the ROI video. Keyframes are densely extracted before the TIC peak and after the peak. Subsequently, based on the difference between two adjacent keyframes, optical flow technology is used to generate an optical flow map. Optical flow features are the displacements of each pixel in the x and y directions between two adjacent frames, and are visualized using grayscale images. In one embodiment, for each video segment, 80 frames are uniformly extracted from the TIC curve's starting point to its peak, and 20 frames are uniformly extracted from the peak to its ending point.

[0030] Step S205: Input the optical flow feature map and the labeled single-frame CEUS image into the dual-flow model to obtain the disease probabilities of five types of diseases, including: the probability of primary hepatocellular carcinoma, the probability of liver metastasis, the probability of intrahepatic cholangiocarcinoma, the probability of hepatic hemangioma, and the probability of liver abscess.

[0031] In the embodiments described in this specification, the two-stream model includes: spatial branching and temporal branching; The step of inputting the optical flow feature map and the labeled single-frame CEUS image into the dual-flow model to obtain the disease probabilities of five types of diseases specifically includes: The optical flow feature map is input into the time branch to obtain the first probability of five types of diseases. The labeled single-frame CEUS image is input into the spatial branch to obtain the second probability of five types of diseases; The disease probabilities of the five types of diseases are obtained by adaptively weighting and fusing the first disease probability and the second disease probability of the five types of diseases.

[0032] In the embodiments of this specification, the step of inputting the optical flow feature map into the time branch to obtain the first probability of five types of diseases specifically includes: The optical flow feature map is input into the time branch. After the convolutional neural network of the time branch extracts the optical flow features of each frame in the optical flow feature map, it is sent into the LSTM network to obtain the time feature vector. The time feature vectors are concatenated with a linear classification head to obtain the first disease probability of the five types of diseases. The structure of the linear classification head includes: a ReLU activation layer, a linear layer, and a Softmax layer. The step of inputting the labeled single-frame CEUS image into the spatial branch to obtain the second disease probability of five types of diseases specifically includes: The labeled single-frame CEUS image is input into the spatial branch, and the spatial features of the labeled single-frame CEUS image are extracted by the convolutional neural network of the spatial branch to obtain the spatial feature vector. By concatenating the spatial feature vectors with the linear classification head, the second probability of the five types of diseases can be obtained.

[0033] In the embodiments described in this specification, the loss function of the two-stream model is a weighted sum of the spatial branch classification loss, the temporal branch classification loss, and the final classification loss; The spatial branch classification loss, the temporal branch classification loss, and the final classification loss are all cross-entropy losses.

[0034] In the embodiments described in this specification, the formula for calculating the loss function of the two-stream model is as follows: ; in, L represents the loss function of the two-stream model. Indicates the spatial branching classification results. This indicates the time-branch classification results. This represents the final branch classification result, and CE represents the loss function.

[0035] , Where j represents the time branch classification result, the spatial branch classification result, or the final branch classification result, respectively.

[0036] Step S207: Input the optical flow feature map into a preset binary sub-model to obtain the positive probabilities of four positive markers, the four positive markers including: Hep, GPC-3, CK7, and CK19.

[0037] In the embodiments of this specification, the preset binary sub-models include: Hep positive vs. Hep negative, GPC3 positive vs. GPC3 negative, CK7 positive vs. CK7 negative, and CK19 positive vs. CK19 negative. The specific method for obtaining the preset binary sub-models can employ existing technologies and is not limited here.

[0038] Step S209: Based on the clinical structured data, perform correlation coefficient screening to obtain clinical feature vectors.

[0039] In the embodiments described in this specification, the clinical structured data includes continuous variables and categorical variables; The process of filtering clinical feature vectors based on the structured clinical data using correlation coefficients specifically includes: The Spearman correlation coefficient was used to calculate the correlation between continuous variables in the clinical structured data and various focal liver lesions (FLL), and the first correlation coefficient for each continuous variable corresponding to various focal liver lesions was obtained. Kendall correlation coefficient was used to calculate the correlation between categorical variables in the clinical structured data and various types of focal liver lesions (FLL), and the second correlation coefficient for each type of focal liver lesion corresponding to each categorical variable was obtained. If the absolute value of the first correlation coefficient of each continuous variable corresponding to each type of focal liver lesion or the absolute value of the second correlation coefficient of each categorical variable corresponding to each type of focal liver lesion is less than a preset threshold, then the variables in the clinical structured data are removed to obtain the clinical feature vector.

[0040] In this embodiment, the continuous variables in the clinical structured data include: age, tumor size, and serological indicators; the categorical variables include: gender, medical history, and pathological information.

[0041] Spearman correlation coefficient, also known as Spearman rank correlation coefficient, is a nonparametric indicator used to measure the dependence between two variables. Kendall correlation coefficient, on the other hand, is a statistical indicator used to measure the ordering consistency between two variables. Spearman correlation coefficient is used to assess the correlation between continuous variables and FLL (Flexible Listed Variable) patterns, while Kendall correlation coefficient is used to assess the correlation between categorical variables and FLL patterns.

[0042] Step S211: After fusing the probabilities of the five types of diseases, the positive probabilities of the four positive markers, and the clinical feature vector, perform OT determination to obtain the classification result of focal liver lesions.

[0043] In this embodiment of the specification, the step of fusing the probabilities of the five types of diseases, the positive probabilities of the four positive markers, and the clinical feature vectors to perform OT determination and obtain the classification result of focal liver lesions specifically includes: The probabilities of the five types of diseases, the positive probabilities of the four positive markers, and the clinical feature vectors are used as inputs to a multilayer perceptron. The processing of the three hidden layers of the multilayer perceptron outputs the classification result of focal liver lesions. If the probability values ​​of primary hepatocellular carcinoma, liver metastases, intrahepatic cholangiocarcinoma, HH, and HA in the classification results of focal liver lesions are all lower than the preset probability values, then it is determined to be OT.

[0044] In the embodiments described in this specification, the preset probability value is 0.2.

[0045] This scheme, through a special design of the MLP network architecture, eliminates the need for additional training of OT vs. non-OT sub-models. Instead, OT is set as a supplementary classification independent of the original five classes within the MLP. The MLP's decision rule can be roughly understood as follows: if the predicted probability of a case in all five classes of FLL is below the corresponding threshold, then the FLL is classified as OT. This design effectively solves the problem of insufficient number and complex types of OT cases, which prevents the training of sub-models based on CEUS videos.

[0046] The liver focal lesion classification method based on contrast-enhanced ultrasound (CEUS) provided in this specification involves acquiring data to be processed, including: CEUS dynamic video and corresponding clinical structured data; extracting keyframes from the CEUS dynamic video, performing optical flow processing on the extracted keyframes to form an optical flow feature map; inputting the optical flow feature map and annotated single-frame CEUS images into a dual-flow model to obtain the disease probabilities of five categories of diseases, including: the probability of primary hepatocellular carcinoma, the probability of liver metastases, the probability of intrahepatic cholangiocarcinoma, the probability of hepatic hemangioma, and the probability of liver abscess; and inputting the optical flow feature map into a preset binary sub-model to obtain four... The positive probability of four positive biomarkers, including Hep, GPC-3, CK7, and CK19, is used to screen the clinical structured data based on correlation coefficients to obtain a clinical feature vector. The probabilities of the five disease categories, the positive probabilities of the four positive biomarkers, and the clinical feature vector are fused together to determine OT (Organic Lesion) and obtain the classification result of focal liver lesions. This method can accurately diagnose multiple types of FLL (Follicular Ligamentary Liver Disease) based on multi-data fusion, overcoming the drawbacks of relying entirely on physician experience. It also features multi-classification capability, cross-vendor stability, low computational requirements, low cost, and ease of use, possessing enormous potential for large-scale clinical application.

[0047] The above describes in detail a method for classifying focal liver lesions based on contrast-enhanced ultrasound. Correspondingly, this specification also provides a classification system for focal liver lesions based on contrast-enhanced ultrasound, such as... Figure 4 As shown. Figure 4 This is a schematic diagram of a liver focal lesion classification system based on ultrasound contrast imaging, provided as an embodiment of this specification. The liver focal lesion classification system includes: Data acquisition module 401 acquires data to be processed, including: CEUS dynamic video and corresponding clinical structured data; The video preprocessing module 403 extracts key frames from the CEUS dynamic video, performs optical flow processing on the extracted key frames, and forms an optical flow feature map. The disease module 405 inputs the optical flow feature map and the labeled single-frame CEUS image into the dual-flow model to obtain the disease probabilities of five types of diseases, including: the probability of primary hepatocellular carcinoma, the probability of liver metastasis, the probability of intrahepatic cholangiocarcinoma, the probability of hepatic hemangioma, and the probability of liver abscess. The pathology module 407 inputs the optical flow feature map into a preset binary sub-model to obtain the positive probabilities of four positive markers, which include: Hep, GPC-3, CK7, and CK19. Clinical module 409 performs correlation coefficient filtering based on the clinical structured data to obtain clinical feature vectors; The feature fusion and classification module 411 fuses the probabilities of the five types of diseases, the positive probabilities of the four positive markers, and the clinical feature vectors, and then performs OT determination to obtain the classification result of focal liver lesions.

[0048] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0049] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments for apparatus, electronic devices, and non-volatile computer storage media are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0050] The apparatus, electronic device, and non-volatile computer storage medium and method provided in the embodiments of this specification are corresponding. Therefore, the apparatus, electronic device, and non-volatile computer storage medium also have similar beneficial technical effects as the corresponding method. Since the beneficial technical effects of the method have been described in detail above, the beneficial technical effects of the corresponding apparatus, electronic device, and non-volatile computer storage medium will not be repeated here.

[0051] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many methodological improvements today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program and "integrate" a digital system onto a PLD themselves, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must also be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0052] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0053] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0054] For ease of description, the above apparatus is described by dividing it into various functional units. Of course, when implementing one or more embodiments of this specification, the functions of each unit can be implemented in one or more software and / or hardware.

[0055] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, the embodiments of this specification can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the embodiments of this specification can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0056] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0057] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0058] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0059] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0060] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0061] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0062] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0063] This specification can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This specification can also be practiced in distributed computing environments, where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside on local and remote computer storage media, including storage devices.

[0064] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0065] The above description is merely an embodiment of this specification and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of this application should be included within the scope of the claims of this application.

Claims

1. A method for classifying focal liver lesions based on contrast-enhanced ultrasound, characterized in that, The classification method for focal liver lesions includes: Acquire the data to be processed, which includes: CEUS dynamic video and corresponding clinical structured data; Key frames of the CEUS dynamic video are extracted, and optical flow processing is performed on the extracted key frames to form an optical flow feature map. The optical flow feature map and the labeled single-frame CEUS image are input into the dual-flow model to obtain the disease probabilities of five types of diseases, including: the probability of primary hepatocellular carcinoma, the probability of liver metastasis, the probability of intrahepatic cholangiocarcinoma, the probability of hepatic hemangioma, and the probability of liver abscess. The optical flow feature map is input into a preset binary sub-model to obtain the positive probabilities of four positive markers, which include: Hep, GPC-3, CK7, and CK19. Based on the aforementioned structured clinical data, correlation coefficients are used for screening to obtain clinical feature vectors; After fusing the probabilities of the five types of diseases, the positive probabilities of the four positive markers, and the clinical feature vectors, OT determination is performed to obtain the classification results of focal liver lesions. The classification results of focal liver lesions include: primary hepatocellular carcinoma, liver metastasis, intrahepatic cholangiocarcinoma, HH, HA, and OT.

2. The method for classifying focal liver lesions as described in claim 1, characterized in that, The step of extracting key frames from the CEUS dynamic video and performing optical flow processing on the extracted key frames to form an optical flow feature map specifically includes: The CEUS dynamic video is annotated to obtain the ROI video; Based on the time intensity curve, key frames are extracted from the ROI video to obtain the extracted key frames; Based on the differences between two adjacent frames of the extracted keyframes, the optical flow feature map is generated using optical flow technology.

3. The method for classifying focal liver lesions as described in claim 2, characterized in that, The step of annotating the CEUS dynamic video to obtain the ROI video specifically includes: The first frame of the CEUS dynamic video is labeled with ROI as a template, and the template branch is input to extract fixed-dimensional lesion template features. The template branch is composed of a CNN backbone network composed of multiple layers of ConvBNReLU modules stacked together. Each frame in the CEUS dynamic video except the first frame is taken as the current frame. The search region image of the current frame is received with the lesion historical location as the center. The search branch is input and the candidate region feature map is extracted. The search branch and the template branch share weights and have the same structure as the CNN backbone network. The template features and the candidate region feature map are cross-correlated and convolved to generate a two-dimensional similarity heatmap. The peak coordinates in the two-dimensional similarity heatmap are used as the initial predicted position of the lesion in the current frame. The initial predicted position of the lesion in the current frame is corrected for the lesion center by time-motion attention, and a rectangular ROI box is generated according to a fixed outer expansion pixel to obtain the lesion ROI image of the current frame. Arrange the lesion ROI images of each current frame in an orderly manner to obtain the ROI video; The process of generating the optical flow feature map using optical flow technology based on the differences between two adjacent frames of the extracted keyframes specifically includes: Based on the two adjacent frames of the extracted keyframe, the pixel-level optical flow field is calculated and the optical flow map is output. The optical flow map includes: horizontal displacement component and vertical displacement component. The optical flow map is corrected by the state equation of a Kalman filter to generate the optical flow feature map.

4. The method for classifying focal liver lesions as described in claim 3, characterized in that, The state equation of the Kalman filter is: ; in, This represents the target's state vector at time t; Represents the state transition matrix; This indicates process noise.

5. The method for classifying focal liver lesions as described in claim 1, characterized in that, The dual-stream model includes: spatial branching and temporal branching; The step of inputting the optical flow feature map and the labeled single-frame CEUS image into the dual-flow model to obtain the disease probabilities of five types of diseases specifically includes: The optical flow feature map is input into the time branch to obtain the first probability of five types of diseases. The labeled single-frame CEUS image is input into the spatial branch to obtain the second probability of five types of diseases; The disease probabilities of the five types of diseases are obtained by adaptively weighting and fusing the first disease probability and the second disease probability of the five types of diseases.

6. The method for classifying focal liver lesions as described in claim 5, characterized in that, The step of inputting the optical flow feature map into the time branch to obtain the first probability of five types of diseases specifically includes: The optical flow feature map is input into the time branch. After the convolutional neural network of the time branch extracts the optical flow features of each frame in the optical flow feature map, it is sent into the LSTM network to obtain the time feature vector. The time feature vectors are concatenated with a linear classification head to obtain the first disease probability of the five types of diseases. The structure of the linear classification head includes: a ReLU activation layer, a linear layer, and a Softmax layer. The step of inputting the labeled single-frame CEUS image into the spatial branch to obtain the second disease probability of five types of diseases specifically includes: The labeled single-frame CEUS image is input into the spatial branch, and the spatial features of the labeled single-frame CEUS image are extracted by the convolutional neural network of the spatial branch to obtain the spatial feature vector. By concatenating the spatial feature vectors with the linear classification head, the second probability of the five types of diseases can be obtained.

7. The method for classifying focal liver lesions as described in claim 5, characterized in that, The loss function of the two-stream model is a weighted sum of the spatial branch classification loss, the temporal branch classification loss, and the final classification loss; The spatial branch classification loss, the temporal branch classification loss, and the final classification loss are all cross-entropy losses.

8. The method for classifying focal liver lesions as described in claim 1, characterized in that, The clinical structured data includes continuous variables and categorical variables; The process of filtering clinical feature vectors based on the structured clinical data using correlation coefficients specifically includes: The Spearman correlation coefficient was used to calculate the correlation between continuous variables in the clinical structured data and various focal liver lesions (FLL), and the first correlation coefficient for each continuous variable corresponding to various focal liver lesions was obtained. Kendall correlation coefficient was used to calculate the correlation between categorical variables in the clinical structured data and various types of focal liver lesions (FLL), and the second correlation coefficient for each type of focal liver lesion corresponding to each categorical variable was obtained. If the absolute value of the first correlation coefficient of each continuous variable corresponding to each type of focal liver lesion or the absolute value of the second correlation coefficient of each categorical variable corresponding to each type of focal liver lesion is less than a preset threshold, then the variables in the clinical structured data are removed to obtain the clinical feature vector.

9. The method for classifying focal liver lesions as described in claim 1, characterized in that, The process of fusing the probabilities of the five disease categories, the positive probabilities of the four positive markers, and the clinical feature vectors to perform OT (Occupational Tract Occlusion) determination and obtain the classification result of focal liver lesions specifically includes: The probabilities of the five types of diseases, the positive probabilities of the four positive markers, and the clinical feature vectors are used as inputs to a multilayer perceptron. The processing of the three hidden layers of the multilayer perceptron outputs the classification result of focal liver lesions. If the probability values ​​of primary hepatocellular carcinoma, liver metastases, intrahepatic cholangiocarcinoma, HH, and HA in the classification results of focal liver lesions are all lower than the preset probability values, then it is determined to be OT.

10. A classification system for focal liver lesions based on contrast-enhanced ultrasound, characterized in that, The classification system for focal liver lesions includes: The data acquisition module acquires the data to be processed, which includes: CEUS dynamic video and corresponding clinical structured data; The video preprocessing module extracts key frames from the CEUS dynamic video, performs optical flow processing on the extracted key frames, and forms an optical flow feature map. The disease module inputs the optical flow feature map and the labeled single-frame CEUS image into the dual-flow model to obtain the disease probabilities of five types of diseases, including: the probability of primary hepatocellular carcinoma, the probability of liver metastasis, the probability of intrahepatic cholangiocarcinoma, the probability of hepatic hemangioma, and the probability of liver abscess. The pathology module inputs the optical flow feature map into a preset binary sub-model to obtain the positive probabilities of four positive markers, which include: Hep, GPC-3, CK7, and CK19. The clinical module performs correlation coefficient filtering based on the aforementioned structured clinical data to obtain clinical feature vectors; The feature fusion and classification module fuses the probabilities of the five types of diseases, the positive probabilities of the four positive markers, and the clinical feature vectors, and then performs OT determination to obtain the classification results of focal liver lesions.

Images