Pathological analysis method and device, electronic equipment and storage medium thereof

Through the multimodal model, the text and picture data are fused with preset disease staging prediction model, the problems of insufficient data utilization and low accuracy in existing pathological analysis methods are solved, and more efficient and accurate pathological analysis is achieved.

CN119069113APending Publication Date: 2024-12-03THE TWELFTH PEOPLE S HOSPITAL AFFILIATED TO GUANGZHOU MEDICAL UNIVERSITY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411163856.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-23
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

Existing pathological analysis methods rely on physician experience and a single data source, resulting in insufficient utilization of analytical data, low accuracy and high cost.

Method used

The preset multimodal model combines text clinical data and picture data, generates feature data, and uses the preset disease staging prediction model for analysis, avoiding natural language tools and manual operations, and improving analysis efficiency and accuracy.

Benefits of technology

It improves the accuracy and efficiency of pathological analysis results, and solves the problems of insufficient data utilization and high cost of manual analysis.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119069113A_ABST
    Figure CN119069113A_ABST
Patent Text Reader

Abstract

Embodiments of the invention provide a pathology analysis method and apparatus, and an electronic device. The method comprises the steps of obtaining target text clinical data and target picture data corresponding to disease categories; performing fusion processing on the target text clinical data and the target picture data through a preset multi-modal model to obtain target feature data of the disease category; and processing the target feature data through a preset disease type staging prediction model to at least obtain a staging result of the disease type. According to the method, the text clinical data and the picture data are fused through the preset multi-modal model to obtain the feature data, the feature data are predicted through the preset disease type stage prediction model, and the corresponding stage number of symptoms of the disease type is determined, so that the pathology of the disease type can be analyzed in combination with the text data and the picture data; and the pathology does not need to be analyzed through a natural language tool or manual conventional operation, so that the efficiency of analyzing the corresponding period number of the pathology is improved, and the accuracy of a pathology analysis result is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of intelligent medicine, and in particular to a pathological analysis method, device, electronic equipment and storage medium thereof. Background Art

[0002] In the medical field, accurate disease staging is an important step in clinical diagnosis and treatment. It helps doctors develop more accurate treatment plans for patients, assess the severity of the disease, and predict the development trend of the disease. In existing pathological analysis technologies, traditional pathological analysis methods mainly rely on the clinical experience of doctors, the clinical manifestations of patients, and limited medical images for analysis. However, these methods have some limitations, such as high dependence on doctor experience, and incomplete information or interpretation errors that may occur on a single data source, which leads to insufficient use of analysis data in existing pathological analysis methods, resulting in low analysis accuracy and high manual analysis costs. Summary of the invention

[0003] The embodiment of the present invention provides a pathology analysis method, which fuses text clinical data and image data through a preset multimodal model to obtain feature data, and predicts the feature data through a preset disease staging prediction model to determine the stage corresponding to the symptoms of the disease. The above method can be used to analyze the pathology of the disease in combination with text data and image data, and there is no need to analyze the pathology through natural language tools or manual routine operations, thereby improving the efficiency of the analysis of the pathology corresponding to the stage, improving the accuracy of the pathology analysis results, and solving the problems in the pathology analysis method, such as insufficient utilization of analysis data, resulting in low analysis accuracy, and high manual analysis cost.

[0004] In a first aspect, an embodiment of the present invention provides a pathological analysis method, the method comprising the following steps: Obtain target text clinical data and target image data for the corresponding disease; The target text clinical data and the target image data are fused and processed by a preset multimodal model to obtain target feature data of the disease; The target characteristic data is processed by a preset disease staging prediction model to at least obtain the staging result of the disease.

[0005] Optionally, before fusing the target text clinical data and the target image data through a preset multimodal model to obtain the target feature data of the disease, the method further includes: Obtain training text clinical data, training image data and multimodal models to be trained for the corresponding diseases; Marking the training text clinical data and the training image data with feature labels to obtain a first training data set; Performing fusion processing on the training text clinical data and training image data marked by the feature labels in the first training data set, so that the training text clinical data and training image data marked by the feature labels are cross-referenced in the same feature space, to obtain a second training data set; The multimodal model to be trained is iteratively trained based on the second training data set, and the preset multimodal model is obtained after the iterative training is completed.

[0006] Optionally, before the target feature data is processed by a preset disease staging prediction model, at least before obtaining the staging result of the disease, the method further includes: Obtaining the staging result label corresponding to the training feature data of the disease and the disease staging prediction model to be trained; Performing staging result prediction processing on the training feature data through the disease staging prediction model to be trained to obtain a test staging result; The disease type staging prediction model to be trained is iteratively trained according to the test staging result and the staging result label, and a preset disease type staging prediction model is obtained after the iterative training is completed.

[0007] Optionally, the target text clinical data and the target image data are fused by a preset multimodal model to obtain the target feature data of the disease, including: Performing multi-dimensional processing on the target text clinical data and the target image data to obtain corresponding target text clinical sub-data and target image sub-data in each dimension; Based on the target text clinical sub-data and the target image sub-data in each dimension, the target feature data of the disease is obtained.

[0008] Optionally, obtaining the target feature data of the disease based on the target text clinical sub-data and the target image sub-data in each dimension includes: Determine the weight value of each dimension based on the lesion characteristics and retention time of the disease in each dimension; Determine target feature sub-data of each dimension based on the target text clinical sub-data and target image sub-data in each dimension; Based on the target characteristic sub-data and the corresponding weight value, the target characteristic data of the disease in the corresponding dimension is determined.

[0009] Optionally, the corresponding disease includes pneumoconiosis, and the target feature data is processed by a preset disease staging prediction model to at least obtain a staging result of the disease, including: The target feature data is processed by a pneumoconiosis staging prediction model to obtain visualized heat map data corresponding to the target feature data; Based on the visualized heat map data, the staging result of pneumoconiosis is determined.

[0010] Optionally, determining the staging result of pneumoconiosis based on the visualized heat map data includes: Determine the probability of each stage of the disease according to the area ratio of the shadow area corresponding to the visualized heat map in the lesion area; Based on the various staging probabilities, the staging results of the corresponding diseases are determined.

[0011] In a second aspect, an embodiment of the present invention further provides a pathology analysis device, the pathology analysis device comprising: The first acquisition module is used to acquire target text clinical data and target image data of the corresponding disease; A second acquisition module is used to fuse the target text clinical data and the target image data through a preset multimodal model to obtain target feature data of the disease; The third acquisition module is used to process the target feature data through a preset disease staging prediction model to at least obtain the staging result of the disease.

[0012] In a third aspect, an embodiment of the present invention provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the pathological analysis method provided in the embodiment of the present invention when executing the computer program.

[0013] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps in the pathological analysis method provided in the embodiment of the invention are implemented.

[0014] In an embodiment of the present invention, target text clinical data and target image data of the corresponding disease are obtained; the target text clinical data and target image data are fused and processed by a preset multimodal model to obtain target feature data of the disease; the target feature data is processed by a preset disease staging prediction model to obtain at least the staging result of the disease. The text clinical data and image data are fused by a preset multimodal model to obtain feature data, and the feature data is predicted by a preset disease staging prediction model to determine the stage number corresponding to the symptoms of the disease. The above method can be used to analyze the pathology of the disease in combination with text data and image data, and there is no need to analyze the pathology through natural language tools or manual routine operations, thereby improving the efficiency of the analysis of the corresponding stage number of the pathology and improving the accuracy of the pathology analysis results. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0016] Figure 1 is a flow chart of a pathological analysis method provided by an embodiment of the present invention; Figure 2 yes Figure 1 The specific flow chart before step 102; Figure 3 yes Figure 1 The specific flow chart before step 103; Figure 4 yes Figure 1 Specific flow chart of step 102; Figure 5 yes Figure 4 Specific flow chart of step 1022; Figure 6 yes Figure 1 The specific flow chart of step 103; Figure 7 yes Figure 6 Specific flow chart of step 1032; Figure 8 is a schematic structural diagram of a pathological analysis device provided in an embodiment of the present invention; Fig. 9 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0017] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0018] like Figure 1 As shown, Figure 1 : is a flow chart of a pathological analysis method provided by an embodiment of the present invention, the pathological analysis method comprising the steps of: 101. Obtain target text clinical data and target image data of the corresponding disease.

[0019] In an embodiment of the present invention, the above-mentioned pathological analysis method can be applied to a pathological analysis platform. The above-mentioned pathological analysis platform has functions such as pathological data processing, pathological data transmission and reception, and pathological data memory storage, and can be constructed based on a server or a server cluster. The above-mentioned server or server cluster can be an electronic device with pathological data processing capabilities.

[0020] The above-mentioned corresponding diseases may include but are not limited to a series of occupational diseases caused by inhalation of dust in the working environment, such as pneumoconiosis, rust lung and byssinosis, and may also be any recorded clinical pathology, such as diseases with clinical characteristics such as rashes, blisters, white spots, etc. It is understandable that the above-mentioned corresponding diseases may have obvious or prominent traits. The above-mentioned traits may be the histological manifestations of the disease, specifically, the observable changes in the structure and morphology of cells and tissues.

[0021] The target text clinical data may be a text description corresponding to the characteristics of the corresponding disease. It is understandable that the target text clinical data may be a narrative text content, such as the chief complaint (including current disease history and past disease history), demographic data (age and gender), and laboratory test results. It should be noted that the demographic data and laboratory test results may be used as structured chief complaint content, while the maximum length of the unstructured chief complaint content may be set to a maximum length threshold, such as 40 characters. In a possible embodiment, if the patient's chief complaint exceeds the maximum length threshold, we only take the chief complaint before the maximum length threshold. If the patient's chief complaint does not exceed the maximum length threshold, zero padding is used to meet the length requirement. The laboratory test results may also be normalized by normalizing the test results of different laboratories for the same disease to obtain the average result of the disease, and use it as the laboratory's target test result for the disease.

[11] The above-mentioned target image data may be the image scan content corresponding to the trait characteristics of the above-mentioned corresponding disease. It can be understood that the target image data can be obtained by scanning and processing the acquired image of the corresponding disease through an image recognition model with pathological image processing, trait feature extraction and trait feature tracking functions.

[0022] In a possible embodiment, the pathology analysis platform processes and analyzes the text and images input by the user or patient, converts them into target text clinical data and target image data, creates files based on the user or patient, and stores them in a database corresponding to the file.

[0023] 102. The target text clinical data and the target image data are fused through a preset multimodal model to obtain the target feature data of the disease.

[0024] In an embodiment of the present invention, the preset multimodal model can be used to fuse and integrate different types of data sources. Specifically, the target text clinical data and target image data can be fused through the preset multimodal model to obtain the target feature data of the disease. The preset multimodal model can include but is not limited to BERT (Bidirectional Encoder Representations from Transformers), CLIP (Contrastive Language–Image Pre-training) and MMBT (Multimodal Bitransformer) and other deep learning models that can fuse the target text clinical data and target image data.

[0025] The above-mentioned target feature data may be key information extracted from the above-mentioned target text clinical data and target image data, and the key information may be used to identify or predict the type of disease. It should be noted that the key information may be a specific shape feature or an indicator value of the shape feature, and the type of key information may be adjusted according to the specific implementation plan to facilitate diagnosis by medical personnel.

[0026] In a possible embodiment, the pathology analysis platform fuses the target text clinical data and the target image data through the preset multimodal model to obtain target feature data corresponding to the disease.

[0027] 103. The target characteristic data is processed by using a preset disease staging prediction model to at least obtain the staging result of the disease.

[0028] In an embodiment of the present invention, the above-mentioned preset disease staging prediction model can be used to determine the development stage of the above-mentioned diseases. Generally speaking, the development stage of the disease can be defined according to the different traits that may occur in the disease at different stages. It should be noted that different diseases may have the same trait characteristics at different stages of development, and in this embodiment, the above-mentioned preset disease staging prediction model can also be used to distinguish the different development stages of different diseases based on the target feature data. The above-mentioned preset disease staging prediction model may include but is not limited to the Inception-DenseNet-V2 network model and any deep learning model that can process the above-mentioned target feature data to determine the staging results of the corresponding disease.

[0029] The above-mentioned staging results can be different development stages of the disease. In a possible embodiment, the above-mentioned pathological analysis platform processes the target feature data through the above-mentioned preset disease staging prediction model to at least obtain the current development stage of the disease, and stores the current development stage as the staging result of the current disease in the database of the corresponding file.

[0030] In an embodiment of the present invention, target text clinical data and target image data of the corresponding disease are obtained; the target text clinical data and target image data are fused and processed by a preset multimodal model to obtain target feature data of the disease; the target feature data is processed by a preset disease staging prediction model to at least obtain the staging result of the disease. The text clinical data and image data are fused by a preset multimodal model to obtain feature data, and the feature data is predicted by a preset disease staging prediction model to determine the stage number corresponding to the symptoms of the disease. The above method can be used to analyze the pathology of the disease in combination with text data and image data, and there is no need to analyze the pathology through natural language tools or manual routine operations, thereby improving the efficiency of the analysis of the corresponding stage number of the pathology and improving the accuracy of the pathology analysis results.

[0031] Optional, such as Figure 2 As shown, before the step 102 of fusing the target text clinical data and the target image data by a preset multimodal model to obtain the target feature data of the disease, steps 104-107 are also included, wherein: 104. Obtain training text clinical data, training image data and the multimodal model to be trained for the corresponding disease.

[0032] In an embodiment of the present invention, the above-mentioned training text clinical data and training image data can be obtained through any channels or institutions that can provide text descriptions corresponding to the characteristics of known diseases, such as text records of clinical studies, public medical databases, and medical academics. It should be noted that the acquisition of training text clinical data and training image data of corresponding diseases by the above-mentioned channels or institutions needs to comply with relevant privacy and legal regulations, and be legally acquired under relevant requirements.

[0033] The multimodal model to be trained may be any deep learning model that has not been trained, such as BERT (Bidirectional Encoder Representations from Transformers), CLIP (Contrastive Language–Image Pre-training), and MMBT (Multimodal Bitransformer), which can fuse the target text clinical data and target image data but cannot accurately output the corresponding target feature data.

[0034] Generally speaking, inputting the training text clinical data and training image data of the corresponding diseases into the multimodal model to be trained may not necessarily output the target feature data of the text clinical data and image data corresponding to the corresponding diseases.

[0035] 105. Perform feature label marking on the training text clinical data and the training image data to obtain a first training data set.

[0036] The above-mentioned feature label marking can be a process of marking descriptions with the same shape content features between the training text clinical data and the training image data. Generally speaking, the above-mentioned feature label marking can be regarded as a process of connecting the training text clinical data and the training image data, that is, expressing the same or similar trait features between the training text clinical data and the training image data with the same content description. For example, when the text content of the training text clinical data describes blisters with convex and large-scale distribution trait features, the pictures corresponding to the trait features with the same convex and large-scale distribution in the training image data can be marked with feature labels, that is, the above-mentioned multimodal model to be trained can match the corresponding picture data of blisters with convex and large-scale distribution trait features through the text corresponding to the above-mentioned blisters with convex and large-scale distribution trait features.

[0037] In a possible embodiment, the above-mentioned pathology analysis platform marks feature labels on training text clinical data and training image data with corresponding content descriptions of the same or similar shape features, so that the training text clinical data and the training image data can be matched, and a first training data set is constructed based on the matched training text clinical data and the corresponding training image data.

[0038] In another possible embodiment, each training image data after feature labeling can be subjected to image enhancement processing by an adaptive histogram equalization algorithm to obtain a plurality of enhanced training image data, and each enhanced training image data is subjected to pixel interval uniform processing to obtain a plurality of interval training image data, and the pixel value of each pixel in each interval training image data is normalized according to the following formula: ; Among them, Meani represents the pixel mean of the i-th image, Stdi represents the pixel standard deviation, IXYZ represents the normalized pixel value, and IXYZ' represents the pixel value before normalization. The above formula can be used to normalize the pixel value of each pixel in each interval training image, thereby enhancing the clarity and accuracy of the interval training image data.

[12] 106. Perform fusion processing on the training text clinical data and the training image data marked by the feature labels in the first training data set, so that the training text clinical data and the training image data marked by the feature labels are cross-referenced in the same feature space to obtain a second training data set.

[0039] The above-mentioned feature space can be a level having the corresponding trait characteristics marked with the above-mentioned feature labels. Generally speaking, in order to enable different types of data to be cross-referenced in the training of the multimodal model to be trained, different types of data can be marked so that different types of data marked with the same feature label can be used for model training at the same trait characteristic level, thereby improving the accuracy of the training model.

[0040] In a possible embodiment, the pathological analysis platform can label the training text clinical data, cross-reference the corresponding training image data and the training text clinical data in the same feature space, and construct the training image data and the training text clinical data according to the feature label to obtain a classifier based on text content, which can be used as the input of the multimodal model to be trained. It should be noted that multiple classifiers can be used as the second training data set.

[0041] 107. Perform iterative training on the multimodal model to be trained based on the second training data set, and obtain a preset multimodal model after the iterative training is completed.

[0042] In a possible embodiment, the second training data set can be divided into a training set, a validation set and a test set. The multimodal model to be trained is trained according to the second training data set, and the parameters of the model are adjusted according to the output results. The performance of the model is optimized through the validation set. Finally, the multimodal model to be trained is evaluated according to the test set. When the multimodal model to be trained processes the input training set and validation set, and the output result is close to the output threshold, it means that the training of the multimodal model to be trained is completed, and the preset multimodal model is obtained.

[0043] Optional, such as Figure 3 As shown, in step 103, the target feature data is processed by a preset disease staging prediction model to at least obtain the staging result of the disease, and steps 108-110 are also included, wherein: 108. Obtain the staging result label corresponding to the training feature data of the disease and the disease staging prediction model to be trained.

[0044] In the embodiment of the present invention, the training characteristic data of the above diseases can be obtained through any channels or institutions that can provide text descriptions of the characteristics of known diseases, such as text records of clinical studies, public medical databases, medical academics, etc. It should be noted that the acquisition of training text clinical data and training image data of the corresponding diseases by the above channels or institutions needs to comply with relevant privacy and legal regulations, and be legally obtained under relevant requirements.

[0045] The above-mentioned staging result labels can be used to accurately match the corresponding characteristic data of the disease at different development stages. That is, the characteristic data is input into the above-mentioned disease staging prediction model to be trained, so as to output the corresponding staging results, and the parameters of the disease staging prediction model to be trained are adjusted through the staging result labels, so as to complete the training process of the disease staging prediction model to be trained.

[0046] The disease staging prediction model to be trained may be an Inception-DenseNet-V2 network model that has not been trained yet, and any deep learning model that can process the target feature data but cannot accurately determine the staging result of the corresponding disease.

[0047] 109. Use the disease staging prediction model to be trained to perform staging result prediction processing on the training feature data to obtain the test staging result.

[0048] The above test staging results may be model output results that cannot accurately match the corresponding disease with corresponding feature data. That is, the test staging results may not match the training feature data. For example, when the test staging result is pneumoconiosis stage 3, the trait characteristics corresponding to the feature data may be pneumoconiosis stage 2 or other trait characteristics that do not match pneumoconiosis stage 3.

[0049] In a possible embodiment, the pathology analysis platform inputs training feature data to the disease staging prediction model to be trained, and evaluates the test staging results output by the disease staging prediction model to be trained to determine whether the disease staging prediction model to be trained has been trained.

[0050] 110. The disease staging prediction model to be trained is iteratively trained according to the test staging results and the staging result labels, and a preset disease staging prediction model is obtained after the iterative training is completed.

[0051] In another possible embodiment, the pathological analysis platform compares the ratio of the test staging result and the staging result label. When the ratio of the test staging result and the staging result label is greater than or equal to the preset training value, it means that the disease staging prediction model to be trained has completed the training, and the preset disease staging prediction model is obtained. It should be noted that the preset training value can be dynamically set according to the incidence rate and retention time of the corresponding disease. Generally speaking, the higher the incidence rate and the longer the retention time of the disease, the higher the requirements and accuracy for diagnosis. Therefore, the setting of the preset training value can be set higher. Generally speaking, the preset training value can be set between 0 and 1.

[0052] Optional, such as Figure 4 As shown, in step 102, the target text clinical data and the target image data are fused by a preset multimodal model to obtain the target feature data of the disease, which also includes steps 1021-1022, wherein: 1021. Perform multi-dimensional processing on the target text clinical data and the target image data to obtain the corresponding target text clinical sub-data and target image sub-data in each dimension.

[0053] In an embodiment of the present invention, the above-mentioned multiple dimensions may refer to multiple data types used for fusion processing, such as text data types or image data types, and may also be feature types in the fusion processing process, such as targeted processing based on the characteristics of the disease onset. For example, fusion processing is performed based on the two characteristics of rapid onset and high diffusion rate. Through this method, data of the two dimensions of rapid onset and high diffusion rate can be fused.

[0054] The above-mentioned target text clinical sub-data can be a text data type with more detailed characteristics obtained after the above-mentioned preset multimodal model performs multi-dimensional processing on the target text clinical data. Specifically, the target text clinical sub-data can be used to describe the characteristics of a disease in the current dimension in a more detailed manner.

[0055] The above-mentioned target image sub-data can be a picture data type with more detailed features obtained after the above-mentioned preset multimodal model performs multi-dimensional processing on the target image data. Specifically, the target image sub-data can be used to display the characteristics of a disease in the current dimension in a more detailed manner.

[0056] 1022. Based on the target text clinical sub-data and target image sub-data in each dimension, the target feature data of the disease is obtained.

[0057] The above-mentioned target feature data can be a numerical value obtained after multi-dimensional processing of the above-mentioned target text clinical data and target image data by the above-mentioned preset multimodal model. In a possible embodiment, the value can be compared with the characteristic values ​​on each dimension. If the above-mentioned target feature data is within the characteristic value interval of the dimension, it means that based on the results of analyzing the target text clinical sub-data and the target image sub-data, it can be judged as the current disease with the characteristics on the current dimension.

[0058] Optional, such as Figure 5 As shown, in step 1022, based on the target text clinical sub-data and the target image sub-data in each dimension, the step of obtaining the target feature data of the disease type also includes steps 10221-10223, wherein: 10221. Determine the weight value of each dimension based on the lesion characteristics and retention time of the disease in each dimension.

[0059] In the embodiment of the present invention, the above-mentioned lesion characteristics can be the trait characteristics or physiological pathological characteristics of the disease in the current dimension, and can be described by text data or image data. For example, it can be the whitening of hair in albinism, or it can be the symptoms of frequent coughing in pneumoconiosis. The above-mentioned lesion characteristics can also be graded according to the degree of damage to the body, and the classification can be determined according to the specific implementation plan. Generally speaking, the relative level of symptoms that appear on the surface, such as whitening of hair and lips, can be set lower, and the symptoms that cause specific damage to the body, such as coughing or lung contraction, can be set higher.

[0060] The above-mentioned retention time can be the length of time that the disease exists in the patient's body. Generally speaking, the retention time can be based on the time period from the initial discovery or diagnosis of the disease to the recovery of the disease.

[0061] The above weight values ​​can be set according to the level of harm to the human body caused by the lesion characteristics and the length of retention time. Generally speaking, in the current dimension, the higher the level of harm corresponding to the feature and the longer the length of time, the higher the corresponding weight value can be set. It should be noted that the overall weight value can be set between 0-1.

[0062] 10222. Based on the target text clinical sub-data and target image sub-data in each dimension, determine the target feature sub-data of each dimension.

[0063] The above-mentioned target feature sub-data can be a numerical value obtained after multi-dimensional processing of the above-mentioned target text clinical sub-data and the target image sub-data by the above-mentioned preset multimodal model. In a possible embodiment, the value can be compared with the feature values ​​on each dimension. If the above-mentioned target feature sub-data is within the feature value range of the dimension, it means that based on the results of analyzing the target text clinical sub-data and the target image sub-data, it can be judged as the current disease with the characteristics on the current dimension.

[0064] 10223. Based on the target feature sub-data and the corresponding weight values, determine the target feature data of the disease in the corresponding dimension.

[0065] In another possible embodiment, the above-mentioned pathology analysis platform processes the target text clinical sub-data and the target image sub-data in a certain dimension through a preset multimodal model to obtain the target feature sub-data corresponding to the target text clinical sub-data and the target image sub-data in the current dimension, and calculates the weight values ​​corresponding to the lesion characteristics and retention time in the current dimension to obtain the target feature data of the disease in the current dimension.

[0066] Optional, such as Figure 6 As shown, in step 103, the target feature data is processed by a preset disease staging prediction model to obtain at least the staging result of the disease, and steps 1031-1032 are also included, wherein: 1031. Process the target feature data through the pneumoconiosis staging prediction model to obtain visualized heat map data corresponding to the target feature data; 1032. Determine the staging results of pneumoconiosis based on visualized heat map data.

[0067] In the embodiment of the present invention, the corresponding diseases may include but are not limited to pneumoconiosis, and the pneumoconiosis staging prediction model belongs to one of the preset disease staging prediction models, and the staging prediction model of the corresponding disease can be selected according to the specific implementation plan. Specifically, according to the disease inferred by the preset multimodal model, the staging prediction model of the corresponding disease can be selected for staging detection.

[0068] The above-mentioned visualized heat map data can be used to determine the lesion area and the possibility of the current disease. Generally speaking, in the visualized heat map data, the darker the color, the greater the possibility that the area is the lesion area, and the brighter the color, the greater the possibility of the current disease. Specifically, since the disease has different characteristics in the early, middle and late stages, and the characteristics are manifested differently in the visualized heat map data, the development stage of the current disease can be predicted based on the results on the visualized heat map data. Generally speaking, the later the development stage, the brighter it will appear on the visualized heat map data, indicating that the disease has existed for a long time and may belong to the middle and late stages.

[0069] In a possible embodiment, the pathology analysis platform processes the target feature data through a staging prediction model corresponding to a specific disease, outputs corresponding visualized heat map data, and predicts the current staging result of the corresponding disease by comparing the staging result of the corresponding disease in the visualized heat map data with the pattern formed by the heat distribution on the visualized heat map data.

[0070] Optional, such as Figure 7 As shown, in step 1032, the step of determining the staging result of pneumoconiosis based on the visualized heat map data also includes steps 10321-10322, wherein: 10321. Determine the probability of each stage of the disease based on the proportion of the shaded area corresponding to the visualized heat map in the lesion area.

[0071] In the embodiment of the present invention, due to the diffusion effect of the disease, by determining the coverage of the disease in the lesion area at the early, middle and late stages, the development stage of the disease can be effectively predicted. Generally speaking, the later the development stage of the disease and the longer the development time, the greater the proportion of the shadow area corresponding to the visualized heat map in the lesion area due to the diffusion effect.

[0072] The above-mentioned staging probabilities can be determined based on the characteristics corresponding to the specific diseases. It is understandable that not all diseases have a spreading effect. For diseases with a spreading effect, the spreading effects at different development stages are also different. Therefore, the staging probabilities of the diseases can be further determined based on the above-mentioned retention time and area proportion. That is, when the area proportion is large and the retention time is short, it means that the disease has the characteristics of a large spreading effect and a fast spreading speed. Therefore, the development stage of the current disease can be determined based on what disease has the characteristics of a large spreading effect and a fast spreading speed at what stage.

[0073] 10322. Based on the probabilities of each stage, determine the staging results of the corresponding disease.

[0074] In a possible embodiment, the pathology analysis platform determines and analyzes the staging probability of the corresponding disease through the ratio of the shadow area corresponding to the visualization heat map to the area of ​​the lesion and the retention time, and determines the staging result of the corresponding disease based on the analysis result.

[0075] like Figure 8 As shown, the embodiment of the present invention further provides a pathological analysis device 800, which includes: The first acquisition module 801 is used to acquire target text clinical data and target image data corresponding to the disease type; The second acquisition module 802 is used to fuse the target text clinical data and the target image data through a preset multimodal model to obtain the target feature data of the disease; The third acquisition module 803 is used to process the target feature data through a preset disease staging prediction model to at least obtain the staging result of the disease.

[0076] Optionally, the above-mentioned pathological analysis device further includes: The fourth acquisition module is used to obtain the training text clinical data, training image data and the multimodal model to be trained corresponding to the disease; A fifth acquisition module is used to perform feature label marking on the training text clinical data and the training image data to obtain a first training data set; a sixth acquisition module, configured to perform fusion processing on the training text clinical data and the training image data marked by the feature labels in the first training data set, so that the training text clinical data and the training image data marked by the feature labels are cross-referenced in the same feature space, and obtain a second training data set; The seventh acquisition module is used to iteratively train the multimodal model to be trained based on the second training data set, and obtain the preset multimodal model after the iterative training is completed.

[0077] Optionally, the above-mentioned pathological analysis device further includes: An eighth acquisition module, used to acquire the staging result label corresponding to the training feature data of the disease and the disease staging prediction model to be trained; A ninth acquisition module, configured to perform staging result prediction processing on the training feature data by using the disease staging prediction model to be trained to obtain a test staging result; The tenth acquisition module is used to iteratively train the disease staging prediction model to be trained according to the test staging results and the staging result labels, and obtain a preset disease staging prediction model after the iterative training is completed.

[0078] Optionally, the second acquisition module 802 includes: A first acquisition submodule is used to perform multi-dimensional processing on the target text clinical data and the target image data to obtain corresponding target text clinical sub-data and target image sub-data in each dimension; The second acquisition submodule is used to obtain the target feature data of the disease based on the target text clinical sub-data and the target image sub-data in each dimension.

[0079] Optionally, the second acquisition submodule includes: A first determination unit is used to determine the weight value of each dimension based on the lesion characteristics and retention time of the disease in each dimension; A second determining unit, configured to determine target feature sub-data of each dimension based on the target text clinical sub-data and the target image sub-data in each dimension; The third determination unit is used to determine the target feature data of the disease in the corresponding dimension based on the target feature sub-data and the corresponding weight value.

[0080] Optionally, the third acquisition module 803 includes: A third acquisition submodule is used to process the target feature data through the pneumoconiosis staging prediction model to obtain visualized heat map data corresponding to the target feature data; The fourth acquisition submodule is used to determine the staging result of pneumoconiosis based on the visualized heat map data.

[0081] Optionally, the fourth acquisition submodule includes: A first determination unit is used to determine the probability of each stage of the disease according to the area ratio of the shadow area corresponding to the visualized heat map in the lesion area; The second determination unit is used to determine the staging result of the corresponding disease based on the various staging probabilities.

[0082] like Fig. 9 As shown, an embodiment of the present invention further provides an electronic device 900, including a processor, and the processor can execute any one of the above-mentioned pathology analysis methods.

[0083] Specifically, it includes a processor 901 and a memory 902, and a computer program for executing the pathological analysis method stored in the memory 902 and capable of running on the processor 901, wherein: The processor 901 runs the computer program of the pathological analysis method stored in the memory 902 and performs the following steps: Obtain target text clinical data and target image data for the corresponding disease; The target text clinical data and the target image data are fused and processed by a preset multimodal model to obtain target feature data of the disease; The target characteristic data is processed by a preset disease staging prediction model to at least obtain the staging result of the disease.

[0084] Optionally, before the processor 901 performs the fusion processing of the target text clinical data and the target image data by using a preset multimodal model to obtain the target feature data of the disease, the method further includes: Obtain training text clinical data, training image data and multimodal models to be trained for the corresponding diseases; Marking the training text clinical data and the training image data with feature labels to obtain a first training data set; Performing fusion processing on the training text clinical data and training image data marked by the feature labels in the first training data set, so that the training text clinical data and training image data marked by the feature labels are cross-referenced in the same feature space, to obtain a second training data set; The multimodal model to be trained is iteratively trained based on the second training data set, and the preset multimodal model is obtained after the iterative training is completed.

[0085] Optionally, before the processor 901 executes the processing of the target feature data by using a preset disease staging prediction model to at least obtain the staging result of the disease, the method further includes: Obtaining the staging result label corresponding to the training feature data of the disease and the disease staging prediction model to be trained; Performing staging result prediction processing on the training feature data through the disease staging prediction model to be trained to obtain a test staging result; The disease type staging prediction model to be trained is iteratively trained according to the test staging result and the staging result label, and a preset disease type staging prediction model is obtained after the iterative training is completed.

[0086] Optionally, the processor 901 performs the fusion processing of the target text clinical data and the target image data through a preset multimodal model to obtain the target feature data of the disease, including: Performing multi-dimensional processing on the target text clinical data and the target image data to obtain corresponding target text clinical sub-data and target image sub-data in each dimension; Based on the target text clinical sub-data and the target image sub-data in each dimension, the target feature data of the disease is obtained.

[0087] Optionally, the processor 901 executes the target text clinical sub-data and the target image sub-data in each dimension to obtain the target feature data of the disease, including: Determine the weight value of each dimension based on the lesion characteristics and retention time of the disease in each dimension; Determine target feature sub-data of each dimension based on the target text clinical sub-data and target image sub-data in each dimension; Based on the target characteristic sub-data and the corresponding weight value, the target characteristic data of the disease in the corresponding dimension is determined.

[0088] Optionally, the processor 901 further executes the corresponding disease including pneumoconiosis, and the processing of the target feature data by using a preset disease staging prediction model to obtain at least a staging result of the disease includes: The target feature data is processed by a pneumoconiosis staging prediction model to obtain visualized heat map data corresponding to the target feature data; Based on the visualized heat map data, the staging result of pneumoconiosis is determined.

[0089] Optionally, the processor 901 further performs the step of determining the staging result of pneumoconiosis based on the visualized heat map data, including: Determine the probability of each stage of the disease according to the area ratio of the shadow area corresponding to the visualized heat map in the lesion area; Based on the various staging probabilities, the staging results of the corresponding diseases are determined.

[0090] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the pathological analysis method or the application-end pathological analysis method provided in the embodiment of the present invention are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0091] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0092] The above disclosure is only the preferred embodiment of the present invention, which certainly cannot be used to limit the scope of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope of the present invention.

Claims

1. A pathological analysis method, characterized in that: include: Obtain target text clinical data and target image data for the corresponding disease; The target text clinical data and the target image data are fused and processed by a preset multimodal model to obtain target feature data of the disease; The target characteristic data is processed by a preset disease staging prediction model to at least obtain the staging result of the disease.

2. The pathological analysis method according to claim 1, characterized in that: Before fusing the target text clinical data and the target image data through a preset multimodal model to obtain the target feature data of the disease, the method further includes: Obtain training text clinical data, training image data and multimodal models to be trained for the corresponding diseases; Marking the training text clinical data and the training image data with feature labels to obtain a first training data set; Performing fusion processing on the training text clinical data and training image data marked by the feature labels in the first training data set, so that the training text clinical data and training image data marked by the feature labels are cross-referenced in the same feature space, to obtain a second training data set; The multimodal model to be trained is iteratively trained based on the second training data set, and the preset multimodal model is obtained after the iterative training is completed.

3. The pathological analysis method according to claim 1, characterized in that: Before the target characteristic data is processed by the preset disease staging prediction model, at least before the staging result of the disease is obtained, the method further includes: Obtaining the staging result label corresponding to the training feature data of the disease and the disease staging prediction model to be trained; Performing staging result prediction processing on the training feature data through the disease staging prediction model to be trained to obtain a test staging result; The disease type staging prediction model to be trained is iteratively trained according to the test staging result and the staging result label, and a preset disease type staging prediction model is obtained after the iterative training is completed.

4. The pathological analysis method according to claim 1, characterized in that: The target text clinical data and the target image data are fused and processed by a preset multimodal model to obtain the target feature data of the disease, including: Performing multi-dimensional processing on the target text clinical data and the target image data to obtain corresponding target text clinical sub-data and target image sub-data in each dimension; Based on the target text clinical sub-data and the target image sub-data in each dimension, the target feature data of the disease is obtained.

5. The pathological analysis method according to claim 4, characterized in that: The target feature data of the disease type is obtained based on the target text clinical sub-data and the target image sub-data in each dimension, including: Determine the weight value of each dimension based on the lesion characteristics and retention time of the disease in each dimension; Determine target feature sub-data of each dimension based on the target text clinical sub-data and target image sub-data in each dimension; Based on the target characteristic sub-data and the corresponding weight value, the target characteristic data of the disease in the corresponding dimension is determined.

6. The pathological analysis method according to claim 1, characterized in that: The corresponding disease includes pneumoconiosis, and the target characteristic data is processed by a preset disease staging prediction model to at least obtain the staging result of the disease, including: The target feature data is processed by a pneumoconiosis staging prediction model to obtain visualized heat map data corresponding to the target feature data; Based on the visualized heat map data, the staging result of pneumoconiosis is determined.

7. The pathological analysis method according to claim 6, characterized in that: Determining the staging result of pneumoconiosis based on the visualized heat map data includes: Determine the probability of each stage of the disease according to the area ratio of the shaded area corresponding to the visualized heat map in the lesion area; Based on the various staging probabilities, the staging results of the corresponding diseases are determined.

8. A pathological analysis device, characterized in that: include: The first acquisition module is used to acquire target text clinical data and target image data of the corresponding disease; A second acquisition module is used to fuse the target text clinical data and the target image data through a preset multimodal model to obtain target feature data of the disease; The third acquisition module is used to process the target feature data through a preset disease staging prediction model to at least obtain the staging result of the disease.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the pathological analysis method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the pathology analysis method according to any one of claims 1 to 7 are implemented.