Liver data processing method and device, computing equipment and storage medium

Abstract

The invention provides a liver data processing method and device, computing equipment and a storage medium. Wherein according to abnormal characteristics in the identified liver image data, feature extraction parameters are dynamically adjusted to extract key image features; according to multilayer semantic information and context information of vocabularies contained in the liver report data, key text features are extracted from the liver report data; determining the mutual influence degree between the key image features and the key text features, and generating fusion features; according to the fusion features, the dependency relationship between different processing tasks and the weight of each processing task are adjusted; and predicting each processing task according to the adjusted weight of each processing task, and outputting a processing result. According to the technical scheme provided by the invention, the processing efficiency and processing effect of the liver data can be improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a liver data processing method, apparatus, computing device, and storage medium. Background Technology

[0002] With advancements in medical imaging technology and clinical documentation, a vast amount of liver imaging data and related clinical text data are collected for medical decision support. Physicians often need to comprehensively analyze this data to develop treatment plans. However, due to the diversity of liver diseases and their complex clinical presentations, relying solely on manual interpretation of this data is often inefficient and lacks high accuracy.

[0003] Current solutions are mostly limited to single-modal data analysis, such as using only imaging data or relying solely on clinical report data, which restricts the comprehensive utilization of information. Furthermore, existing technologies often fail to effectively integrate these two types of data; that is, the interaction between imaging data and clinical text data is not fully utilized. For example, some solutions may only focus on extracting abnormal features from imaging data without considering the supplementary information that may be provided in the clinical text, which may lead to less than ideal processing results and poor performance in liver data processing. Summary of the Invention

[0004] This application provides a liver data processing method, apparatus, computing device, and storage medium to solve the problem of poor liver data processing performance in the prior art.

[0005] In a first aspect, embodiments of this application provide a liver data processing method, including:

[0006] Obtain the user's liver imaging data and liver report data;

[0007] Based on the identified abnormalities in the liver imaging data, the feature extraction parameters are dynamically adjusted, and key imaging features are extracted from the liver imaging data using the feature extraction parameters.

[0008] Based on the multi-layered semantic information and contextual information of the words contained in the liver report data, key text features are extracted from the liver report data;

[0009] Determine the degree of mutual influence between the key image features and the key text features, and generate fused features;

[0010] Based on the fusion characteristics, the dependencies between different processing tasks and the weight of each processing task are adjusted;

[0011] Based on the adjusted weights of each processing task, a prediction is made for each processing task, and the processing result is output.

[0012] Optionally, determining the degree of mutual influence between the key image features and the key text features, and generating fused features, includes:

[0013] Determine the first attention weight matrix of the key image features to the key text features, and the second attention weight matrix of the key text features to the key image features, respectively;

[0014] Based on the first attention matrix and the second attention matrix, the adjusted key image features and adjusted key text features are determined.

[0015] Based on the adjusted key image features and the adjusted key text features, a fusion feature is generated.

[0016] Optionally, the step of dynamically adjusting the feature extraction parameters based on the identified abnormalities in the liver imaging data, and extracting key imaging features from the liver imaging data using the feature extraction parameters, includes:

[0017] Data of multiple abnormal regions contained in the liver imaging data are obtained, so as to identify the abnormal characteristics of the user in the abnormal region through the data of each abnormal region. The abnormal characteristics include at least: region area, region complexity, and region resolution.

[0018] Based on the abnormal characteristics of the user in multiple abnormal regions, the feature extraction parameters are dynamically adjusted, wherein each abnormal characteristic of an abnormal region corresponds to a feature extraction parameter.

[0019] Intermediate image features are extracted from the liver image data according to each of the aforementioned feature extraction parameters, and multiple intermediate image features are fused to obtain key image features.

[0020] Optionally, the step of extracting key text features from the liver report data based on the multi-layered semantic information and contextual information of the words contained in the liver report data includes:

[0021] The liver report data was segmented into multiple words and converted into corresponding word vectors;

[0022] The lexical meaning of each word vector is determined by a multi-layer semantic aggregation algorithm, and the semantic information between multiple word vectors is determined layer by layer, so as to determine the multi-layer semantic information of the words contained in the liver report data.

[0023] By using a pre-trained large language model, the contextual information between multiple word vectors is determined;

[0024] Based on the multi-layered semantic information and the contextual information, key text features are extracted from the liver report data.

[0025] Optionally, determining the first attention weight matrix of the key image features to the key text features, and the second attention weight matrix of the key text features to the key image features, respectively, includes:

[0026] The first attention weight matrix is ​​calculated based on the query vector of the key image feature and the key vector of the key text feature.

[0027] The second attention weight matrix is ​​calculated based on the query vector of the key text features and the key vector of the key image features.

[0028] The step of determining the adjusted key image features and adjusted key text features based on the first attention matrix and the second attention matrix includes:

[0029] The value vectors of the key text features are weighted and summed using the first attention weight matrix to obtain the adjusted key image features.

[0030] The value vectors of the key image features are weighted and summed using the second attention weight matrix to obtain the adjusted key text features.

[0031] Optionally, adjusting the dependencies between different processing tasks and the weight of each processing task based on the fusion features includes:

[0032] Calculate the dependencies between different processing tasks in the fusion features;

[0033] The weights of each processing task are updated based on the fusion features and the dependencies between different processing tasks.

[0034] Optionally, after acquiring the user's liver imaging data and liver report data, the method further includes:

[0035] The liver imaging data is standardized, and the standardization process includes at least: segmenting the target liver data from the liver imaging data, reducing noise in the liver imaging data, and adjusting the intensity values ​​of the liver imaging data to the same range;

[0036] The liver report data is preprocessed, and the preprocessing includes at least: segmenting the liver report data into multiple words and converting the words in the liver report data into corresponding word vectors.

[0037] Secondly, embodiments of this application provide a liver data processing apparatus, comprising:

[0038] The acquisition module is used to acquire the user's liver imaging data and liver report data;

[0039] The adjustment module is used to dynamically adjust the feature extraction parameters based on the abnormal characteristics identified in the liver imaging data;

[0040] The extraction module is used to extract key image features from the liver image data using the feature extraction parameters; and to extract key text features from the liver report data based on the multi-layer semantic information and contextual information of the words contained in the liver report data.

[0041] The determination module is used to determine the degree of mutual influence between the key image features and the key text features, and to generate fused features;

[0042] The adjustment module is also used to adjust the dependencies between different processing tasks and the weight of each processing task according to the fusion features;

[0043] The output module is used to predict each of the processing tasks based on the adjusted weights of each processing task, and output the processing results.

[0044] Thirdly, embodiments of this application provide a computing device, including a processing component and a storage component; the storage component stores one or more computer instructions; the one or more computer instructions are to be invoked and executed by the processing component to implement the liver data processing method as described in the first aspect above.

[0045] Fourthly, embodiments of this application provide a computer storage medium storing a computer program. When the computer program is executed by a computer, it implements the liver data processing method described in the first aspect above. In this application embodiment, liver imaging data and liver report data of a user are acquired; based on identified abnormal characteristics in the liver imaging data, feature extraction parameters are dynamically adjusted, and key imaging features are extracted from the liver imaging data using the feature extraction parameters; key text features are extracted from the liver report data based on multi-layer semantic information and contextual information of the words contained in the liver report data; the degree of mutual influence between the key imaging features and the key text features is determined, and a fusion feature is generated; based on the fusion feature, the dependencies between different processing tasks and the weight of each processing task are adjusted; based on the adjusted weight of each processing task, prediction is performed on each processing task, and the processing result is output.

[0046] This application's solution effectively extracts key imaging and textual features by acquiring users' liver imaging and report data. The solution dynamically adjusts feature extraction parameters based on anomalies in the liver imaging data, thereby improving the accuracy and relevance of feature extraction. By analyzing the mutual influence between key imaging and textual features, fused features are generated, which contributes to a more comprehensive understanding of liver condition. Adjusting the dependencies between different processing tasks and the weight of each task based on the fused features makes the entire processing workflow more flexible and efficient, better adaptable to the complex and ever-changing needs of medical data processing. Predictions are made based on the adjusted processing task weights, and the processing results are output, which helps improve the processing efficiency and effectiveness of liver data.

[0047] These or other aspects of this application will become more apparent in the following description of the embodiments. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 A flowchart of a liver data processing method provided in an embodiment of this application;

[0050] Figure 2 This is a schematic diagram of the structure of a liver data processing device provided in an embodiment of this application;

[0051] Figure 3 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application. Detailed Implementation

[0052] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0053] In some of the processes described in the specification, claims, and accompanying drawings of this application, multiple operations appearing in a specific order are included. However, it should be clearly understood that these operations may not be executed in the order they appear herein, or may be executed in parallel. The operation numbers, such as 101, 102, etc., are merely used to distinguish different operations and do not themselves represent any execution order. Furthermore, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should be noted that the descriptions such as "first," "second," etc., in this document are used to distinguish different messages, devices, modules, etc., and do not represent a chronological order, nor do they limit "first" and "second" to different types.

[0054] The inventors' research revealed that current liver data processing solutions are mostly limited to single-modal data analysis, such as using only imaging data or relying solely on clinical report data, which restricts the comprehensive utilization of information. Furthermore, existing solutions often fail to effectively integrate these two types of data; that is, the interaction between imaging data and clinical text data is not fully utilized. For example, some solutions may only focus on extracting abnormal features from imaging data without considering the supplementary information that may be provided in the clinical text, which may lead to less than ideal processing results and poor liver data processing effectiveness.

[0055] In view of this, embodiments of this application provide a liver data processing method, which includes: acquiring a user's liver imaging data and liver report data; dynamically adjusting feature extraction parameters based on identified abnormal characteristics in the liver imaging data, and extracting key imaging features from the liver imaging data using the feature extraction parameters; extracting key text features from the liver report data based on multi-layer semantic information and contextual information of the words contained in the liver report data; determining the degree of mutual influence between the key imaging features and the key text features, and generating fusion features; adjusting the dependencies between different processing tasks and the weight of each processing task based on the fusion features; predicting each processing task based on the adjusted weight of each processing task, and outputting the processing result.

[0056] This application's solution effectively extracts key imaging and textual features by acquiring users' liver imaging and report data. The solution dynamically adjusts feature extraction parameters based on anomalies in the liver imaging data, thereby improving the accuracy and relevance of feature extraction. By analyzing the mutual influence between key imaging and textual features, fused features are generated, which contributes to a more comprehensive understanding of liver condition. Adjusting the dependencies between different processing tasks and the weight of each task based on the fused features makes the entire processing workflow more flexible and efficient, better adaptable to the complex and ever-changing needs of medical data processing. Predictions are made based on the adjusted processing task weights, and the processing results are output, which helps improve the processing efficiency and effectiveness of liver data.

[0057] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0058] Figure 1 A flowchart of a liver data processing method is provided as an embodiment of this application, such as... Figure 1 As shown, this method is processed and executed by the system;

[0059] The method includes:

[0060] 101. Obtain the user's liver imaging data and liver report data;

[0061] In this step, acquiring the user's liver imaging data refers to obtaining images of the liver region from various imaging devices, such as computed tomography (CT) machines and magnetic resonance imaging (MRI) machines. This imaging data is typically in digital format, such as image files under the Digital Imaging and Communications in Medicine (DICOM) standard.

[0062] Obtaining liver report data from users refers to liver-related examination report documents prepared by doctors or medical technicians. These reports typically include descriptions of imaging data, observed abnormalities, and other clinical information.

[0063] In this embodiment of the application, the liver imaging data and liver report data of the user can be obtained automatically through the hospital information system or manually uploaded.

[0064] For example, to obtain liver imaging data and liver report data for user A, when user A undergoes a liver CT scan, a series of DICOM format image files are obtained. These files are automatically uploaded to the hospital's PACS system and transmitted to the processing system of this application through a specific interface. Furthermore, based on user A's liver CT examination, physician B prepares a report. The report states: "A low-density shadow with a diameter of approximately 2 cm is visible in the right lobe of the liver," and provides other clinical information. This report is stored as a text file in the hospital's EMR system and is obtained by the processing system of this application through the system interface.

[0065] Alternatively, user A can upload liver imaging data and liver report data to the processing system of this application, and the processing system of this application can perform subsequent data processing steps based on the liver imaging data and liver report data.

[0066] 102. Based on the identified abnormal features in the liver imaging data, dynamically adjust the feature extraction parameters, and extract key imaging features from the liver imaging data using the feature extraction parameters;

[0067] Optionally, in this embodiment of the application, step 102 may specifically include:

[0068] 1021. Obtain data of multiple abnormal regions contained in the liver imaging data, so as to identify the abnormal characteristics of the user corresponding to the abnormal region through the data of each abnormal region;

[0069] In this step, an abnormal region refers to a region in liver imaging data that differs from normal tissue. Abnormal characteristics refer to specific attributes of the abnormal region, including at least: region area, region complexity, and region resolution.

[0070] In this context, region area refers to the size or volume of the anomalous region. Region complexity refers to the structural complexity within the anomalous region, such as the irregularity of its edges and variations in its internal texture. Region resolution refers to the sharpness of the anomalous region within the image.

[0071] 1022. Based on the abnormal characteristics of the user in multiple abnormal regions, dynamically adjust the feature extraction parameters;

[0072] Specifically, step 102 can be performed using an Adaptive Convolutional Neural Network (AdaptiveCNN). The feature extraction parameters can be the convolution kernels of AdaptiveCNN, which are small matrices used for feature extraction in AdaptiveCNN. Their parameters include size, number, and stride.

[0073] In step 1022, the feature extraction parameters can be dynamically adjusted based on the characteristics of the anomaly. For example, for smaller anomaly regions, AdaptiveCNN can use smaller convolutional kernels to capture detailed features. For larger anomaly regions, it may use larger convolutional kernels to capture broader features. For anomaly regions with high complexity, AdaptiveCNN can increase the number of convolutional kernels to capture more feature combinations. Conversely, it can reduce the number of convolutional kernels to reduce computational burden. For anomaly regions requiring high resolution, AdaptiveCNN can use smaller strides to preserve more details. Conversely, it can use larger strides to accelerate feature extraction.

[0074] It should be noted that each abnormal region corresponds to a specific feature extraction parameter based on its abnormal characteristics. For example, if abnormal region A has a large area, high complexity, and high resolution, then the feature extraction parameters used will be those adapted to abnormal region A (i.e., including the appropriate kernel size, number, and stride). In contrast, if abnormal region B has a smaller area, then other feature extraction parameters adapted to it will be used (i.e., the kernel size, number, and stride used will be different from those used for abnormal region A).

[0075] 1023. Extract intermediate image features from the liver image data according to each of the aforementioned feature extraction parameters, and fuse multiple intermediate image features to obtain key image features.

[0076] In this step, intermediate image features refer to preliminary features extracted from liver imaging data. These features will be fused in subsequent steps to generate key image features. Key image features are comprehensive features obtained by fusing multiple intermediate image features, which more comprehensively reflect abnormalities in liver imaging.

[0077] Here is a specific example:

[0078] Equipment: Liver scan was performed using a 3.0T magnetic resonance imaging system.

[0079] Data format: The acquired liver imaging data are in DICOM standard format.

[0080] Implementation process: Multiple abnormal regions in liver images are identified using an automatic segmentation algorithm, and data for each abnormal region is extracted. For example, low-density regions in liver images are identified as abnormal regions using a threshold segmentation method.

[0081] Step 1: Obtain data on multiple abnormal regions contained in the liver imaging data, so as to identify the abnormal characteristics of the user corresponding to the abnormal region through the data of each abnormal region;

[0082] In this case, it is assumed that two abnormal regions were identified in the liver MRI image, one in the right lobe of the liver and the other in the left lobe of the liver.

[0083] The abnormal characteristics are as follows:

[0084] Region size: The abnormal region in the right lobe of the liver is 30 square millimeters, and the abnormal region in the left lobe of the liver is 20 square millimeters.

[0085] Regional complexity: The abnormal region in the right lobe of the liver has relatively smooth edges and low complexity; the abnormal region in the left lobe of the liver has irregular edges and high complexity.

[0086] Region resolution: The two abnormal regions have the same resolution, which is 0.5mm × 0.5mm.

[0087] Step 2: Based on the abnormal characteristics of the user in multiple abnormal regions, dynamically adjust the feature extraction parameters. For example, for regions with high edge smoothness, a smaller convolution kernel size can be used to reduce edge blurring.

[0088] For example, for the abnormal region in the right lobe of the liver, the corresponding feature extraction parameters are adjusted as follows:

[0089] Kernel size: 3×3;

[0090] Number of convolution kernels: 32;

[0091] Step size: 1;

[0092] For example, for the abnormal region in the left lobe of the liver, the corresponding feature extraction parameters are adjusted as follows:

[0093] Kernel size: 5×5;

[0094] Number of convolution kernels: 64;

[0095] Step size: 2;

[0096] Step 3: Extract intermediate image features from the liver image data according to each of the feature extraction parameters, and fuse multiple intermediate image features to obtain key image features;

[0097] For example, the intermediate image features obtained for an abnormal region in the right lobe of the liver are shown below:

[0098] Edge features: High edge sharpness;

[0099] Texture characteristics: The texture is relatively uniform;

[0100] For example, the intermediate image features obtained for the abnormal region in the left lobe of the liver are shown below:

[0101] Edge characteristics: Low edge sharpness;

[0102] Texture characteristics: The texture is relatively complex;

[0103] Furthermore, the intermediate image features of the right and left lobes of the liver are fused to obtain key image features. Optionally, a weighted average method can be used to determine the weights based on the size and complexity of each abnormal region.

[0104] Through the above embodiments, we can see how to dynamically adjust feature extraction parameters based on the characteristics of abnormal areas in liver imaging data, and extract key imaging features from the liver imaging data, thereby providing necessary input for subsequent processing steps. This processing method is not limited to the liver, but can also be applied to other organs or medical imaging data.

[0105] 103. Extract key text features from the liver report data based on the multi-layered semantic information and contextual information of the words contained in the liver report data;

[0106] Optionally, in this embodiment of the application, step 103 may specifically include:

[0107] 1031. The liver report data is segmented into multiple words and converted into corresponding word vectors;

[0108] In this step, word segmentation refers to dividing the text in the liver report data into individual words. Word vectors refer to converting each word into a mathematical vector to represent the meaning of that word.

[0109] In this embodiment of the application, the liver report data is as follows: "The liver is of normal size with smooth edges and no obvious abnormal structures are observed inside, but a low-density shadow with a diameter of about 2 cm is found in the right lobe of the liver."

[0110] After segmenting the liver report data into multiple terms, the term list is as follows: ["liver", "size", "normal", "margin", "smooth", "internal", "not seen", "obvious", "abnormal", "structure", "but", "in", "right lobe of liver", "found", "one", "diameter", "approximately", "for", "2", "cm", "of", "low density shadow"]

[0111] Furthermore, pre-trained word embedding models (such as Word to Vector (Word2Vec) and Global Vectors for Word Representation (GloVe)) are used to convert each word into a corresponding word vector.

[0112] Word vector example:

[0113] The word vector for "liver" is: [0.1, 0.3, -0.2, ...].

[0114] The word vector for "abnormal" is: [-0.4, 0.6, 0.1, ...].

[0115] 1032. By using a multi-layer semantic aggregation algorithm, the lexical meaning of each word vector is determined, and the semantic information between multiple word vectors is determined layer by layer, so as to determine the multi-layer semantic information of the words contained in the liver report data;

[0116] In this step, a multi-layer semantic aggregation algorithm is used to determine the semantic information between word vectors layer by layer, thereby revealing the deeper meaning in the text.

[0117] In this embodiment of the application, the multi-layer semantic aggregation algorithm uses models such as Recurrent Neural Network (RNN) or Long Short-Term Memory (LSTM) to aggregate semantic information between word vectors layer by layer.

[0118] The semantic aggregation process is as follows:

[0119] First layer: Determine the basic meaning of each word.

[0120] The second layer: Determine the relationship between adjacent words, such as the relationship between "liver" and "size".

[0121] The third level: Determine the meaning at the sentence level, such as the meaning of "normal liver size" as a whole.

[0122] The process of determining the semantic information between multiple word vectors layer by layer to determine the multi-layer semantic information of the vocabulary contained in the liver report data is as follows:

[0123] First layer of semantic information:

[0124] The basic meaning of "liver" is: to refer to an organ in the human body.

[0125] The basic meaning of "abnormal" is: to deviate from the normal state.

[0126] Second layer of semantic information:

[0127] The relationship between "liver" and "abnormal": describing the state of the liver.

[0128] Third-level semantic information:

[0129] The meaning of "normal liver size": It describes that the size of the liver is within the normal range.

[0130] 1033. By using a pre-trained large language model, determine the contextual information between multiple word vectors;

[0131] In this step, the pre-trained large language model refers to a pre-trained deep learning model used to understand contextual information in text.

[0132] In this embodiment of the application, the pre-trained large language model uses a pre-trained Transformer model (such as Bidirectional Encoder Representations from Transformers, BERT) or a Robustly Optimized BERT Pretraining Approach (RoBERTa) to understand the contextual information between word vectors.

[0133] For example, the contextual information "liver size is normal" indicates that the liver size is normal, which is a positive indicator of overall health. The contextual information "a low-density shadow with a diameter of approximately 2 cm was found in the right lobe of the liver" describes an abnormality found in a specific area of ​​the liver, requiring further attention.

[0134] 1034. Based on the multi-layer semantic information and the contextual information, extract key text features from the liver report data.

[0135] In this step, key text features refer to the most important and meaningful textual information extracted from liver report data.

[0136] In this embodiment, key text features are extracted by combining multi-layered semantic information and contextual information. The following are examples of the extracted key text features:

[0137] The key textual feature "normal liver size" indicates that the liver size is normal.

[0138] The key textual feature "a low-density shadow with a diameter of about 2 cm was found in the right lobe of the liver" indicates that an abnormal area was found in the right lobe of the liver, which requires further attention.

[0139] The above examples demonstrate how to extract key textual features from liver report data, providing necessary input for subsequent processing steps. This processing method is not limited to liver reports but can also be applied to other medical report data.

[0140] 104. Determine the degree of mutual influence between the key image features and the key text features, and generate fused features.

[0141] Optionally, in this embodiment of the application, step 104 may specifically include:

[0142] 1041. Determine the first attention weight matrix of the key image features to the key text features, and the second attention weight matrix of the key text features to the key image features;

[0143] Optionally, a first attention weight matrix is ​​calculated based on the query vector of the key image feature and the key vector of the key text feature; a second attention weight matrix is ​​calculated based on the query vector of the key text feature and the key vector of the key image feature.

[0144] In this embodiment of the application, the attention weight matrix calculation formula is used: A IT =softmax(Q) I *K T T ) Calculate the first attention weight matrix of key image features to key text features, where Q I It is the query vector of key image features, K T It is the key vector of key text features, K T T Represented as K T The transpose of .

[0145] The attention weight matrix is ​​calculated using the following formula: A TI =softmax(Q) T *K I T ) Calculate the second attention weight matrix of the key text features to the key image features, where Q T It is the query vector of key text features, K I It is the key vector of key image features, K I T Represented as K I The transpose of .

[0146] The query vector is a vector extracted from key image features, used to query relevant information from key text features. The key vector is a vector extracted from key text features, used to respond to the query vector's request. The attention weight matrix is ​​obtained by calculating the dot product (inner product) between the query vector and the key vector, and then normalizing it using the softmax function. This matrix reflects the degree of attention given to key text features by key image features.

[0147] 1042. Determine the adjusted key image features and adjusted key text features based on the first attention matrix and the second attention matrix;

[0148] Optionally, the value vectors of the key text features are weighted and summed using the first attention weight matrix to obtain the adjusted key image features; the value vectors of the key image features are weighted and summed using the second attention weight matrix to obtain the adjusted key text features.

[0149] In this embodiment of the application, attention weight matrix A is used. IT and A TI The key image features and key text features are weighted and summed to obtain the adjusted feature I. Star and T Star .

[0150] Specifically, I Star =A IT *V T V T It is the value vector of key textual features. T Star =A TI *V I V I It is the value vector of key image features.

[0151] 1043. Generate fused features based on the adjusted key image features and the adjusted key text features.

[0152] In this application embodiment, the generated fusion features may include, but are not limited to: splicing, weighted summation, element-wise multiplication. This application does not limit these features, and they can be set according to specific needs.

[0153] For example, taking image stitching as an example, suppose we have already obtained: the adjusted key image features I Star = [0.2, 0.3, 0.5]; Adjusted key text features T Star = [0.4, 0.6, 0.8], then the fusion feature F = [0.2, 0.3, 0.5; 0.4, 0.6, 0.8].

[0154] 105. Based on the fusion features, adjust the dependencies between different processing tasks and the weight of each processing task;

[0155] Optionally, in this embodiment of the application, step 105 may specifically include:

[0156] 1051. Calculate the dependencies between different processing tasks in the fusion feature;

[0157] This step involves analyzing the interdependencies between the various processing tasks within the fused features. This dependency refers to whether the successful execution of one task depends on the outcome or state of another task. Dependencies can be determined by calculating the similarity, correlation, or causal relationship between features from different tasks. In this embodiment, the processing tasks are for different types of liver image features and text features, rather than general image recognition and text understanding tasks.

[0158] In this embodiment, a multimodal Transformer system is considered, designed to handle multiple tasks related to liver image features and text features. The fused features incorporate comprehensive information from both the image and text.

[0159] For example, the following processing tasks are included:

[0160] Task 1: Abnormal area detection (identifying abnormal areas in liver images).

[0161] Task 2: Anomaly Description Matching (matching anomaly regions with anomaly features in text descriptions).

[0162] Task 3: Anomaly Severity Assessment (Assess the severity of the anomaly region).

[0163] Furthermore, the Pearson Correlation Coefficient (PCC) can be used to quantify the dependencies between different tasks. The PCC value ranges from -1 to 1, where 1 represents a perfect positive correlation, -1 represents a perfect negative correlation, and 0 represents no correlation.

[0164] Suppose we have a fused feature F that contains features relevant to different tasks. The dependencies between these feature vectors can be estimated by calculating the Pearson correlation coefficient between them.

[0165] For example, through the formula: Calculate the dependencies between different processing tasks;

[0166] Among them, T i and T j Let T represent the feature vectors of the i-th and j-th tasks, respectively. i,kand T j,k Let i and j represent the feature values ​​of the i-th and j-th tasks of the k-th sample, respectively. and These represent the average values ​​of these characteristics.

[0167] Here is a specific example:

[0168] Suppose we have the following data:

[0169] The anomaly region detection feature T1 = [0.2, 0.3, 0.5];

[0170] The anomaly description matching feature T2 = [0.4, 0.6, 0.8];

[0171] The severity assessment feature for the abnormality is T3 = [0.1, 0.2, 0.3].

[0172] These features are extracted from the fused feature F and represent features of different tasks.

[0173] First, calculate their average:

[0174]

[0175] Then, calculate the Pearson correlation coefficient according to the above formula:

[0176]

[0177] This means that there is a high positive correlation between Task 1 (abnormal region detection) and Task 2 (abnormal description matching), a low correlation between Task 1 and Task 3, and a high positive correlation between Task 2 and Task 3.

[0178] 1052. Update the weight of each processing task based on the fusion features and the dependencies between different processing tasks.

[0179] In this step, once the dependencies between tasks are determined, the weight of each task can be adjusted based on these relationships. Adjusting the weights reflects the importance and priority of the tasks, thereby optimizing the overall system performance. The goal of weight updates is to better balance the needs of different tasks, enabling the system to achieve its overall objectives more efficiently.

[0180] In this embodiment of the application, based on the example of step 1051 above, the task weights need to be adjusted according to the dependencies. Here we can use a simple linear model to update the weights, where the magnitude of the dependencies directly affects the weight allocation.

[0181] Suppose we have a basic weight vector w = [w1, w2, w3], where w1, w2, and w3 represent the initial weights of task 1, task 2, and task 3, respectively. These weights can be adjusted based on dependencies so that the weight vector w′ better reflects the interactions between tasks.

[0182]

[0183] Here, α represents a hyperparameter used to control the degree of influence of dependencies on weights.

[0184] Here is a specific example:

[0185] Assume the following initial weights:

[0186] w1 = 0.3;

[0187] w2 = 0.3;

[0188] w3 = 0.4;

[0189] Assume α = 0.1, and the previously calculated Pearson correlation coefficients are PCC(T1,T2) = 0.89, PCC(T1,T3) = 0.45, and PCC(T2,T3) = 0.71.

[0190] Based on the formula above, we get:

[0191]

[0192] Therefore, the updated weights are:

[0193] w′1=0.434

[0194] w′2=0.46

[0195] w′3=0.516

[0196] This means that the weight of Task 3 (anomaly severity assessment) has been slightly increased, while the weights of Task 1 (anomaly region detection) and Task 2 (anomaly description matching) have also been adjusted accordingly, reflecting the strong correlation between these tasks. This weight adjustment helps to better balance the needs of these tasks in subsequent processing.

[0197] 106. Based on the adjusted weights of each processing task, predict each processing task and output the processing result.

[0198] In this step, the processing tasks include, but are not limited to, processing tasks for different types of liver imaging features and text features, such as abnormal region detection, abnormal description matching, and abnormal severity assessment.

[0199] The processing results include, but are not limited to, probability distributions of different types of liver features. These probability distributions can be used to represent the likelihood of a particular feature occurring, helping us to better understand and interpret the processing results.

[0200] Adjusted weights are used to guide the prediction process for each processing task. Each processing task is performed on different types of liver imaging and text features. In this way, we can ensure that the processing tasks better reflect the interdependencies between them, thereby improving the overall prediction accuracy.

[0201] In this embodiment, the previously mentioned processing tasks are continued: anomaly region detection, anomaly description matching, and anomaly severity assessment. It is assumed that an adjusted weight vector w′ = [0.434, 0.46, 0.516] is already available, where 0.434 corresponds to the weight of the anomaly region detection task, 0.46 corresponds to the weight of the anomaly description matching task, and 0.516 corresponds to the weight of the anomaly severity assessment task.

[0202] The task prediction process includes the following steps:

[0203] Anomaly detection: Anomaly detection models (which can be convolutional neural networks or other machine learning models) are used to predict the probability of the existence of anomaly regions.

[0204] Anomaly description matching: Use an anomaly description matching model (which can be a Transformer-based model or other text matching model) to predict the degree of matching between anomaly descriptions and anomaly regions.

[0205] Anomaly Severity Assessment: An anomaly severity assessment model (which can be a regression model or other assessment model) is used to predict the severity of anomaly regions.

[0206] The output processing results include the following steps:

[0207] For each task, a probability distribution or an evaluation value is obtained as the output. Here, it is assumed that the output of the anomaly region detection task is a probability distribution of a binary classification problem, the output of the anomaly description matching task is also a probability distribution of a binary classification problem, and the output of the anomaly severity evaluation task is a continuous value.

[0208] Here is a specific example:

[0209] Suppose we have the following data:

[0210] The output probability distribution of the anomaly detection model is p1 = [0.8, 0.2], where the first value represents the probability of the anomaly region existing and the second value represents the probability of it not existing.

[0211] The output probability distribution of the anomaly description matching model is p2 = [0.7, 0.3], where the first value represents the probability of a description match and the second value represents the probability of a mismatch.

[0212] The output of the anomaly severity assessment model is s = 0.6, which represents the severity of the anomaly region.

[0213] The corresponding processing results are output as follows:

[0214] Anomaly detection: The probability of an anomaly being present is 0.8.

[0215] Anomaly description matching: The probability of a description matching is 0.7.

[0216] Anomaly severity assessment: The severity assessment value for the anomaly area is 0.6.

[0217] These processing results can then be used for subsequent decision-making or analysis. For example, based on the severity assessment, abnormal areas can be categorized into different levels (e.g., mild, moderate, severe) to facilitate the development of more specific management strategies. For instance, if the abnormal description matching results show a high degree of agreement, the accuracy of the description can be further verified and added to the final report as reference information. Alternatively, combining the results of abnormal area detection can clarify the location and extent of the abnormal area, which is crucial for subsequent treatment planning. Furthermore, decision-making or analysis can also be included; this application does not limit this, and specific details can be set according to requirements.

[0218] The technical solution provided in this application effectively extracts key imaging and textual features by acquiring the user's liver imaging data and liver report data. This solution dynamically adjusts feature extraction parameters based on anomalies in the liver imaging data, thereby improving the accuracy and relevance of feature extraction. By analyzing the degree of interaction between key imaging and textual features, fused features are generated, which helps to more comprehensively understand the liver condition. Adjusting the dependencies between different processing tasks and the weight of each task based on the fused features makes the entire processing flow more flexible and efficient, better adapting to the complex and ever-changing needs of medical data processing. Predicting based on the adjusted processing task weights and outputting the processing results helps improve the processing efficiency and effectiveness of liver data.

[0219] In some embodiments, after step 101 "acquiring the user's liver imaging data and liver report data", the method further includes:

[0220] The liver imaging data is standardized, and the standardization process includes at least: segmenting the target liver data from the liver imaging data, reducing noise in the liver imaging data, and adjusting the intensity values ​​of the liver imaging data to the same range.

[0221] This step primarily involves the concept of "standardization," the purpose of which is to make liver imaging data more consistent and easier to process. Standardization mainly includes the following aspects:

[0222] Segmenting target liver data refers to extracting the liver region from the raw image, typically achieved using image segmentation algorithms such as convolutional neural network models like U-Net. This helps remove the influence of background and other tissues, making subsequent processing more accurate.

[0223] Reducing noise in liver imaging data refers to using filters or other noise reduction techniques to reduce random variations in the image and improve image quality. Commonly used filters include Gaussian filters and median filters.

[0224] Adjusting the intensity values ​​of liver imaging data to a similar range refers to using normalization or standardization techniques to ensure that images from different sources have similar gray levels or pixel intensities, thereby improving image consistency and comparability. Commonly used normalization methods include min-max normalization and z-score standardization.

[0225] In this embodiment, the user's liver imaging data and liver report data are first acquired. Next, the liver imaging data undergoes standardization processing, with the specific steps as follows:

[0226] Segmenting target liver data: The liver region is segmented from the liver image data using an automatic segmentation algorithm based on U-Net.

[0227] Reduce noise in liver imaging data: Use a median filter to denoise the segmented liver region, and let the denoised liver region be Lfiltered.

[0228] Adjust the intensity values ​​of liver imaging data to the same range: use the minimum-maximum normalization method to normalize the pixel intensity values ​​of Lfiltered to between 0 and 1.

[0229] In some embodiments, after step 101 "acquiring the user's liver imaging data and liver report data", the method further includes:

[0230] The liver report data is preprocessed, and the preprocessing includes at least: segmenting the liver report data into multiple words and converting the words in the liver report data into corresponding word vectors.

[0231] In this step, tokenization refers to the process of dividing text data into a series of words or sub-words. This is a fundamental step in natural language processing, which helps with subsequent word vector conversion and semantic analysis.

[0232] Word embeddings are fixed-length vector representations of words in text, capturing the semantic and syntactic relationships between words. Common word embedding models include Word-to-Vector (Word2Vec) and Global Vectors for Word Representation (GloVe).

[0233] In this embodiment, the user's liver imaging data and liver report data are first acquired. Next, the liver report data is preprocessed, with the following specific steps:

[0234] The liver report data is segmented into multiple words: word segmentation techniques are used to divide the text data into a series of words or sub-words. For example, spaces, punctuation marks, etc. can be used as delimiters, or more complex word segmentation tools, such as the Natural Language Toolkit (NLTK), can be used.

[0235] The words in the liver report data are converted into corresponding word vectors: pre-trained word vector models (such as Word2Vec, GloVe, etc.) are used to convert the segmented words into vector form. These vectors can be used for subsequent text analysis and processing.

[0236] Figure 2 This application provides a schematic diagram of the structure of a liver data processing device, as shown in the embodiment. Figure 2 As shown, the device includes:

[0237] Module 21 is used to acquire the user's liver imaging data and liver report data;

[0238] Adjustment module 22 is used to dynamically adjust feature extraction parameters based on the abnormal characteristics identified in the liver image data;

[0239] Extraction module 23 is used to extract key image features from the liver image data using the feature extraction parameters; and to extract key text features from the liver report data based on the multi-layer semantic information and contextual information of the words contained in the liver report data.

[0240] The determination module 24 is used to determine the degree of mutual influence between the key image features and the key text features, and to generate fused features;

[0241] The adjustment module 22 is also used to adjust the dependencies between different processing tasks and the weight of each processing task according to the fusion features;

[0242] The output module 25 is used to predict each of the processing tasks according to the adjusted weight of each processing task, and output the processing result.

[0243] Optionally, in this embodiment of the application, the determining module 24 is specifically used to determine the first attention weight matrix of the key image features to the key text features and the second attention weight matrix of the key text features to the key image features; determine the adjusted key image features and the adjusted key text features based on the first attention matrix and the second attention matrix; and generate fused features based on the adjusted key image features and the adjusted key text features.

[0244] Optionally, in this embodiment of the application, the adjustment module 22 is specifically used to acquire data of multiple abnormal regions contained in the liver imaging data, so as to identify the abnormal characteristics of the user corresponding to the abnormal region through the data of each abnormal region. The abnormal characteristics include at least: region area, region complexity, and region resolution; and dynamically adjust the feature extraction parameters according to the abnormal characteristics of the user corresponding to the multiple abnormal regions, wherein each abnormal characteristic corresponding to the abnormal region corresponds to a feature extraction parameter.

[0245] The extraction module 23 is specifically used to extract intermediate image features from the liver image data according to each of the feature extraction parameters, and to fuse multiple intermediate image features to obtain key image features.

[0246] Optionally, in this embodiment, the extraction module 23 is further configured to segment the liver report data into multiple words and convert them into corresponding word vectors; determine the meaning of each word vector through a multi-layer semantic aggregation algorithm, and determine the semantic information between multiple word vectors layer by layer, so as to determine the multi-layer semantic information of the words contained in the liver report data; determine the context information between multiple word vectors through a pre-trained large language model; and extract key text features from the liver report data based on the multi-layer semantic information and the context information.

[0247] Optionally, in this embodiment, the determining module 24 is specifically configured to: calculate a first attention weight matrix based on the query vector of the key image feature and the key vector of the key text feature; calculate a second attention weight matrix based on the query vector of the key text feature and the key vector of the key image feature; use the first attention weight matrix to perform a weighted summation on the value vector of the key text feature to obtain the adjusted key image feature; and use the second attention weight matrix to perform a weighted summation on the value vector of the key image feature to obtain the adjusted key text feature.

[0248] Optionally, in this embodiment of the application, the adjustment module 22 is further configured to calculate the dependency relationship between different processing tasks in the fusion feature; and update the weight of each processing task according to the fusion feature and the dependency relationship between different processing tasks.

[0249] Optionally, in this embodiment of the application, the device further includes a preprocessing module 26;

[0250] The preprocessing module 26 is used to perform standardization processing on the liver image data. The standardization processing includes at least: segmenting the target liver data from the liver image data, reducing noise in the liver image data, and adjusting the intensity values ​​of the liver image data to the same range.

[0251] Optionally, in this embodiment of the application, the preprocessing module 26 is further configured to preprocess the liver report data, the preprocessing including at least: segmenting the liver report data into multiple words and converting the words in the liver report data into corresponding word vectors.

[0252] Figure 2 The liver data processing device described above can perform... Figure 1 The implementation principle and technical effects of the liver data processing method described in the illustrated embodiments will not be repeated here. The specific methods by which each module and unit of the liver data processing device in the above embodiments performs its operations have been described in detail in the embodiments related to this method, and will not be elaborated upon here.

[0253] In one possible design, Figure 2 The liver data processing device shown in the embodiment can be implemented as a computing device, such as... Figure 3 As shown, the computing device may include a storage component 31 and a processing component 32;

[0254] The storage component 31 stores one or more computer instructions, wherein the one or more computer instructions are invoked and executed by the processing component 32.

[0255] The processing component 32 is used to: acquire the user's liver imaging data and liver report data; dynamically adjust feature extraction parameters based on the identified abnormal characteristics in the liver imaging data, and extract key imaging features from the liver imaging data using the feature extraction parameters; extract key text features from the liver report data based on the multi-layer semantic information and contextual information of the words contained in the liver report data; determine the degree of mutual influence between the key imaging features and the key text features, and generate fusion features; adjust the dependencies between different processing tasks and the weight of each processing task based on the fusion features; predict each processing task based on the adjusted weight of each processing task, and output the processing result.

[0256] The processing component 32 may include one or more processors to execute computer instructions to complete all or part of the steps in the above-described method. Alternatively, the processing component may be implemented as one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above-described method.

[0257] Storage component 31 is configured to store various types of data to support operations at the terminal. The storage component can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0258] Of course, computing devices may also include other components, such as input / output interfaces, display components, communication components, etc.

[0259] Input / output interfaces provide interfaces between processing components and peripheral interface modules, which can be output devices, input devices, etc.

[0260] The communication components are configured to facilitate wired or wireless communication between computing devices and other devices.

[0261] The computing device can be a physical device or an elastic computing host provided by a cloud computing platform. In this case, the computing device can refer to a cloud server, and the aforementioned processing components, storage components, etc., can be basic server resources rented or purchased from the cloud computing platform.

[0262] This application also provides a computer storage medium storing a computer program, which, when executed by a computer, can perform the above-described functions. Figure 1 The liver data processing method shown in the embodiment.

[0263] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0264] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0265] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0266] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A liver data processing method, characterized in that, include: Obtain the user's liver imaging data and liver report data; Based on the identified abnormalities in the liver imaging data, the feature extraction parameters are dynamically adjusted, and key imaging features are extracted from the liver imaging data using the feature extraction parameters. Based on the multi-layered semantic information and contextual information of the words contained in the liver report data, key text features are extracted from the liver report data; Determine the degree of mutual influence between the key image features and the key text features, and generate fused features; Based on the fusion characteristics, the dependencies between different processing tasks and the weight of each processing task are adjusted; Based on the adjusted weights of each processing task, a prediction is made for each processing task, and the processing result is output.

2. The method according to claim 1, characterized in that, The step of determining the degree of mutual influence between the key image features and the key text features, and generating fused features, includes: Determine the first attention weight matrix of the key image features to the key text features, and the second attention weight matrix of the key text features to the key image features, respectively; Based on the first attention matrix and the second attention matrix, the adjusted key image features and adjusted key text features are determined. Based on the adjusted key image features and the adjusted key text features, a fusion feature is generated.

3. The method according to claim 1, characterized in that, The step of dynamically adjusting feature extraction parameters based on the identified abnormalities in the liver imaging data, and extracting key imaging features from the liver imaging data using the feature extraction parameters, includes: Data of multiple abnormal regions contained in the liver imaging data are obtained, so as to identify the abnormal characteristics of the user in the abnormal region through the data of each abnormal region. The abnormal characteristics include at least: region area, region complexity, and region resolution. Based on the abnormal characteristics of the user in multiple abnormal regions, the feature extraction parameters are dynamically adjusted, wherein each abnormal characteristic of an abnormal region corresponds to a feature extraction parameter. Intermediate image features are extracted from the liver image data according to each of the aforementioned feature extraction parameters, and multiple intermediate image features are fused to obtain key image features.

4. The method according to claim 1, characterized in that, The step of extracting key text features from the liver report data based on multi-layered semantic information and contextual information of the vocabulary contained in the liver report data includes: The liver report data was segmented into multiple words and converted into corresponding word vectors; The lexical meaning of each word vector is determined by a multi-layer semantic aggregation algorithm, and the semantic information between multiple word vectors is determined layer by layer, so as to determine the multi-layer semantic information of the words contained in the liver report data. By using a pre-trained large language model, the contextual information between multiple word vectors is determined; Based on the multi-layered semantic information and the contextual information, key text features are extracted from the liver report data.

5. The method according to claim 2, characterized in that, The step of determining the first attention weight matrix of the key image features to the key text features, and the second attention weight matrix of the key text features to the key image features, includes: The first attention weight matrix is ​​calculated based on the query vector of the key image feature and the key vector of the key text feature. The second attention weight matrix is ​​calculated based on the query vector of the key text features and the key vector of the key image features. The step of determining the adjusted key image features and adjusted key text features based on the first attention matrix and the second attention matrix includes: The value vectors of the key text features are weighted and summed using the first attention weight matrix to obtain the adjusted key image features. The value vectors of the key image features are weighted and summed using the second attention weight matrix to obtain the adjusted key text features.

6. The method according to claim 1, characterized in that, The step of adjusting the dependencies between different processing tasks and the weight of each processing task based on the fusion features includes: Calculate the dependencies between different processing tasks in the fusion features; The weights of each processing task are updated based on the fusion features and the dependencies between different processing tasks.

7. The method according to claim 1, characterized in that, After acquiring the user's liver imaging data and liver report data, the process also includes: The liver imaging data is standardized, and the standardization process includes at least: segmenting the target liver data from the liver imaging data, reducing noise in the liver imaging data, and adjusting the intensity values ​​of the liver imaging data to the same range; The liver report data is preprocessed, and the preprocessing includes at least: segmenting the liver report data into multiple words and converting the words in the liver report data into corresponding word vectors.

8. A liver data processing device, characterized in that, include: The acquisition module is used to acquire the user's liver imaging data and liver report data; The adjustment module is used to dynamically adjust the feature extraction parameters based on the abnormal characteristics identified in the liver imaging data; The extraction module is used to extract key image features from the liver image data using the feature extraction parameters; and to extract key text features from the liver report data based on the multi-layer semantic information and contextual information of the words contained in the liver report data. The determination module is used to determine the degree of mutual influence between the key image features and the key text features, and to generate fused features; The adjustment module is also used to adjust the dependencies between different processing tasks and the weight of each processing task according to the fusion features; The output module is used to predict each of the processing tasks based on the adjusted weights of each processing task, and output the processing results.

9. A computing device, characterized in that, It includes a processing component and a storage component; the storage component stores one or more computer instructions; the one or more computer instructions are invoked and executed by the processing component to implement the liver data processing method as described in any one of claims 1 to 7.

10. A computer storage medium, characterized in that, The device contains a computer program that, when executed by a computer, implements the liver data processing method as described in any one of claims 1 to 7.

Images