Case analysis method and system based on deep learning

By integrating image and text data and using deep learning methods for case analysis, the problem that a single data type cannot fully reflect case information is solved, the accuracy and completeness of case analysis is improved, and the representation ability of the model is enhanced.

CN119028602AActive Publication Date: 2024-11-26JIANGSU COLLEGE OF NURSING
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202411174935.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-26
Publication Date
2024-11-26
Estimated Expiration
2044-08-26

AI Technical Summary

Technical Problem

Existing case analysis methods mainly rely on a single type of data and cannot fully reflect the entire information of the case. The image data cannot capture medical history information, and the text data may miss key visual clues, resulting in inefficient analysis and insufficient accuracy.

Method used

Using a deep learning-based method, the image data and text data are integrated, image feature extraction is performed through pre-trained convolutional neural network, text data is analyzed in combination with natural language processing models, and new feature vectors are generated using linear interpolation fusion to expand the data set and Training deep learning models for case analysis.

Benefits of technology

It improves the accuracy and completeness of case analysis, overcomes the limitations of single modal information, enhances the model characterization ability, and improves the accuracy and generalization ability of case analysis.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119028602A_ABST
    Figure CN119028602A_ABST
Patent Text Reader

Abstract

The invention provides a case analysis method and system based on deep learning. The method comprises the following steps: acquiring a target data set containing image data and text data; performing feature extraction on the image data in the target data set based on a pre-trained convolutional neural network; inputting the extracted feature vector into a pre-trained natural language processing model for analyzing text data in the target data set; according to the image feature vector processed by the convolutional neural network and the text feature vector processed by the natural language processing model, generating a new feature vector through linear interpolation fusion; expanding the target data set based on the generated new feature vector, and training a deep learning model based on the expanded target data set to obtain a trained deep learning model; and inputting a to-be-analyzed case as input into the trained deep learning model to obtain a case analysis result output by the trained deep learning model.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of medical information processing, and specifically, to a case analysis method and system based on deep learning. Background Art

[0002] In recent years, with the explosive growth of medical data, how to efficiently process and utilize this data has become a research focus. Traditional case analysis mainly relies on doctors' experience and judgment. Although this method is intuitive and reliable, it is often inefficient when faced with a large amount of case data and is prone to human errors. Therefore, it is particularly important to develop automated and intelligent case analysis methods.

[0003] Most existing case analysis methods focus on a single type of data, such as image data or text data, and rarely use both types of data for comprehensive analysis. For example, for the analysis of medical images, models such as convolutional neural networks (CNN) are usually used for feature extraction; while for the analysis of medical record texts, natural language processing (NLP) technology, such as pre-trained models such as BERT, are more often used. However, using any one data type alone cannot fully reflect all the information of the case. For example, relying solely on image data may not capture important text information such as medical history, while relying solely on text data may miss key visual clues.

[0004] Therefore, the present application provides a case analysis method and system based on deep learning to solve one of the above-mentioned technical problems. Summary of the invention

[0005] The purpose of this application is to provide a case analysis method and device based on deep learning, which can solve at least one of the technical problems mentioned above. The specific solution is as follows:

[0006] According to the specific implementation of the present application, in the first aspect, the present application provides a case analysis method based on deep learning, comprising:

[0007] Acquire a target data set including image data and text data; wherein the image data and the text data are respectively obtained based on material cases; perform feature extraction on the image data in the target data set based on a pre-trained convolutional neural network; input the extracted feature vector into a pre-trained natural language processing model for analyzing the text data in the target data set; generate new feature vectors through linear interpolation fusion based on the image feature vectors processed by the convolutional neural network and the text feature vectors processed by the natural language processing model; expand the target data set based on the generated new feature vectors, and train a deep learning model based on the expanded target data set to obtain a trained deep learning model; input the case to be analyzed as an input into the trained deep learning model to obtain a case analysis result output by the trained deep learning model.

[0008] In one embodiment, the convolutional neural network is a pre-trained ResNet50 model.

[0009] In one embodiment, the ResNet50 model adopts the following settings: the preprocessing box of the ResNet50 model is configured with a standardization module, the data input to the preprocessing box is processed by the standardization module and output as a pixel value with normalized characteristics, and the value range of the pixel value conforms to 0 to 1; the ResNet50 model is configured with a rotation module, a scaling module, a flipping module, and a randomization module, the randomization module performs random and repeatable extraction and parallel arrangement in the rotation module, the scaling module, and the flipping module to obtain a module array; the module array is used to output image feature vectors with different generalization degrees based on multiple different output heads matching the number of columns; the ResNet50 model is configured with an architecture fine-tuning module, which is used to increase or decrease the number of convolution layers according to the medical image characteristics of the case to be analyzed; wherein the randomization module performs random and repeatable extraction and parallel arrangement based on the number of convolution layers to make the number of convolution layers consistent with the number of rows of the module array.

[0010] In one embodiment, the natural language processing model is a pre-trained BERT model.

[0011] In one embodiment, the BERT model adopts the following settings: the BERT model performs primary pre-training based on general text data and secondary pre-training based on medical field text data; wherein the medical field text data is configured with a customized word segmenter based on a medical vocabulary; the BERT model is configured with a network layer dynamic connection module, which is used to query the matching fully connected layer and output layer according to the medical field when the medical field to which the case to be analyzed belongs is determined based on text recognition, and the queried fully connected layer and output layer are set as layer connections according to the preconfigured connection method; the BERT model is configured with a loop termination module, which is used to terminate the loop when the loop round of the BERT model reaches the round threshold, and instruct the dynamic connection module to reset the layer connection according to the upper-level medical field of the medical field to which the case to be analyzed belongs that is currently identified; the BERT model is configured with a startup module, which is used to start the loop when the layer connection is set.

[0012] In one embodiment, the linear interpolation fusion includes: selecting feature vectors of two different cases to be analyzed in the target data set as reference points; calculating a linear interpolation coefficient based on the distance between the two feature vectors; and generating a new feature vector based on the linear interpolation coefficient.

[0013] In one embodiment, the expansion of the target data set includes: fusing feature maps of image data of the same dimension in the target data set through improved linear interpolation, and reconstructing them back to image space through a decoder to generate a new medical image; and expanding the text data in the target data set based on the new medical image processed by improved linear interpolation in combination with a pre-trained natural language processing model.

[0014] In one implementation, the linear interpolation formula includes: mixed =α·F A +β·F B ; Among them, F mixed represents the result after interpolation, F A 、F B They respectively represent the feature maps of image data of the same dimension, α represents the interpolation parameter, and β represents the nonlinear degree control parameter.

[0015] In one implementation, the linear interpolation formula includes: mixed =α′·F A +β′·F B ; Among them, F mixed represents the result after interpolation, F A 、F B They represent the feature maps of image data of the same dimension, α′ and β′ represent the optimized interpolation parameters, which are obtained using the following formula: Among them, C represents the complexity factor of the case to be analyzed, T represents the weight of the case type to be analyzed, D represents the time weight, and α and β represent the basic difference parameters respectively.

[0016] According to the specific implementation of the present application, in the second aspect, the present application provides a case analysis system based on deep learning, characterized in that it includes:

[0017] A data acquisition module is used to acquire a target data set including image data and text data, wherein the image data and the text data are respectively obtained based on material cases; an image feature extraction module is used to perform feature extraction on the image data in the target data set based on a pre-trained convolutional neural network; a text feature analysis module is used to input the extracted feature vector into a pre-trained natural language processing model for analyzing the text data in the target data set; a feature fusion module is used to generate a new feature vector by linear interpolation fusion based on the image feature vector processed by the convolutional neural network and the text feature vector processed by the natural language processing model; a data set expansion module is used to expand the target data set based on the generated new feature vector, and train a deep learning model based on the expanded target data set to obtain a trained deep learning model; an input-output module is used to input the case to be analyzed as an input into the trained deep learning model to obtain the case analysis result output by the trained deep learning model.

[0018] Compared with the prior art, the above scheme of the embodiment of the present application has at least the following beneficial effects: The present application provides a case analysis method and system based on deep learning. The method integrates image data and text data. On the one hand, it uses a pre-trained convolutional neural network to extract image features, which can effectively capture key information in medical images. On the other hand, it uses a pre-trained natural language processing (NLP) model to analyze text data, automatically understand the semantics and contextual information in the medical record text, and then analyze the case from multiple angles to improve the accuracy and completeness of the case analysis. This method overcomes the limitations of single modality information, enhances the model representation capability, and thus makes the case analysis based on deep learning more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 A flowchart of a case analysis method based on deep learning according to an embodiment of the present application is shown;

[0020] Figure 2 A flow chart of a linear interpolation fusion method is shown;

[0021] Figure 3 A flow chart of a method for expanding a target data set is shown;

[0022] Figure 4 A module block diagram of a deep learning-based case analysis system according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0024] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application. The singular forms "a", "said" and "the" used in the embodiments of the present application and the appended claims are also intended to include plural forms, and unless the context clearly indicates other meanings, "multiple" generally includes at least two.

[0025] It should be understood that the term "and / or" used herein is only a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0026] It should be understood that although the terms first, second, third, etc. may be used to describe in the embodiments of the present application, these descriptions should not be limited to these terms. These terms are only used to distinguish the descriptions. For example, without departing from the scope of the embodiments of the present application, the first may also be referred to as the second, and similarly, the second may also be referred to as the first.

[0027] Depending on the context, the words "if" and "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting." Similarly, depending on the context, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)."

[0028] It should also be noted that the term "includes", "comprising" or any other variation thereof is intended to cover non-exclusive inclusion, so that a commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprising a ..." do not exclude the existence of other identical elements in the commodity or device including the elements.

[0029] It should be particularly noted that any symbols and / or numbers in the specification that are not marked in the accompanying drawings are not drawing marks.

[0030] The optional embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0031] The embodiment provided in this application is an embodiment of a case analysis method based on deep learning.

[0032] Combine the following Figure 1 The embodiments of the present application are described in detail.

[0033] Figure 1 A flowchart of a case analysis method based on deep learning according to an embodiment of the present application is shown, as Figure 1 As shown, the process includes the following steps S101 to S106.

[0034] In step S101 , a target data set including image data and text data is obtained.

[0035] Among them, the image data and text data are data obtained based on the material cases.

[0036] In step S102, feature extraction is performed on the image data in the target data set based on the pre-trained convolutional neural network.

[0037] In step S103, the extracted feature vector is input into a pre-trained natural language processing model for analyzing text data in the target data set.

[0038] In step S104, a new feature vector is generated by linear interpolation fusion based on the image feature vector processed by the convolutional neural network and the text feature vector processed by the natural language processing model.

[0039] In step S105, the target data set is expanded based on the generated new feature vector, and a deep learning model is trained based on the expanded target data set to obtain a trained deep learning model.

[0040] In step S106, the case to be analyzed is taken as input and input into the trained deep learning model to obtain the case analysis result output by the trained deep learning model.

[0041] The present application provides a case analysis method and system based on deep learning. The method integrates image data and text data, and on the one hand, uses a pre-trained convolutional neural network to extract image features, which can effectively capture key information in medical images. On the other hand, it uses a pre-trained natural language processing (NLP) model to analyze text data, automatically understand the semantics and contextual information in the medical record text, and then analyze the case from multiple angles to improve the accuracy and completeness of the case analysis. This method overcomes the limitations of single modal information, enhances the model representation capability, and thus makes the case analysis based on deep learning more accurate.

[0042] In this application, the target data set can be expanded based on the generated new feature vectors. Based on this approach, on the one hand, the problem of data scarcity in the medical field can be effectively alleviated, and on the other hand, the expanded data set helps to improve the diversity and robustness of model training, thereby improving the generalization ability of the model.

[0043] In one embodiment, the convolutional neural network may be a pre-trained ResNet50 model.

[0044] As a feasible implementation, the ResNet50 model adopts the following ResNet50 model setting one to ResNet50 model setting three.

[0045] ResNet50 model setting 1: The preprocessing box of the ResNet50 model is configured with a normalization module. The data input to the preprocessing box is processed by the normalization module and output as pixel values ​​with normalized features. The value range of the pixel values ​​conforms to 0 to 1.

[0046] In this application, the purpose of setting one of the ResNet50 model is to standardize the input image data, ensure that all image data have the same scale range, and eliminate the impact caused by differences in original image brightness, contrast, etc. Based on this setting, on the one hand, the stability and convergence speed of model training can be improved, and the problem of data gradient disappearance or explosion can be avoided. On the other hand, the model can pay more attention to the structural information of the image rather than the absolute intensity value, which helps to improve the generalization ability of the model.

[0047] ResNet50 model setting 2: The ResNet50 model is configured with a rotation module, a scaling module, a flipping module, and a randomization module. The randomization module randomly and repeatedly extracts and arranges in parallel in the rotation module, the scaling module, and the flipping module to obtain a module array. The module array is used to output image feature vectors of different generalization degrees based on multiple different output heads matching the number of columns.

[0048] In this application, the purpose of ResNet50 model setting 2 is to increase the diversity of the data set by performing a series of enhancement operations on the image data, such as rotation, scaling, flipping, etc., to simulate more possible clinical scenarios, and to select and arrange the enhancement operations using a randomization module to generate image feature vectors with different degrees of generalization. Based on this setting, the robustness of the model to image changes can be further improved, so that it can still maintain good performance when facing images of different angles, sizes or directions. In addition, the data output of multiple different output heads can play a role in expanding the data set, thereby reducing the risk of overfitting and improving the performance of the model on new data.

[0049] ResNet50 model setting 3: The ResNet50 model is equipped with an architecture fine-tuning module, which is used to increase or decrease the number of convolutional layers according to the characteristics of the medical images of the cases to be analyzed. Among them, the randomization module performs random and repeatable extraction based on the number of convolutional layers and arranges them in parallel so that the number of convolutional layers is consistent with the number of rows in the module array.

[0050] In this application, the purpose of setting three of the ResNet50 model is to adjust the depth of the ResNet50 model according to a specific type of medical image, such as increasing or decreasing the number of convolutional layers, and in some embodiments, also includes a personalized setting of the convolutional layer structure. In this application, a randomization module is used to select the number of convolutional layers and ensure that it is consistent with the number of rows of the module array. Based on this setting, it is possible to meet the needs of adaptive data processing and analysis for cases in different medical fields. For example, for case images in some specific medical fields, richer detailed descriptions are required, and then a deeper network is used to capture subtle features. Correspondingly, for some specific medical fields that do not focus on the analysis of case images (for example, more analysis is required through text information), coarse data can be provided by reducing the number of convolutional layers and simplifying the convolutional layer structure, reducing unnecessary computational complexity, avoiding overfitting, and ensuring that the model can learn enough features.

[0051] For example, the natural language processing model can be a pre-trained BERT model.

[0052] As a feasible implementation, the BERT model may adopt the following BERT model settings 1 to BERT model settings 4.

[0053] BERT model setting 1: The BERT model is pre-trained on general text data for the first level and on medical text data for the second level. The medical text data is configured with a customized word segmenter based on a medical vocabulary.

[0054] In this application, the purpose of setting 1 of the BERT model is to perform a first-level pre-training through general text data so that the model can capture a wide range of language structures and patterns, and to perform a second-level pre-training through medical field text data so that the model can better understand the professional terms and context in the medical field. Based on this setting, on the one hand, the model's ability to understand medical texts is enhanced, and its applicability and accuracy in the medical field are improved. On the other hand, the customized word segmenter can process medical-specific vocabulary more accurately, further improving model performance.

[0055] BERT model setting 2: The BERT model is configured with a network layer dynamic connection module, which is used to determine the medical field to which the case to be analyzed belongs based on text recognition, query the matching fully connected layer and output layer according to the medical field, and set the queried fully connected layer and output layer as layer connections according to the preconfigured connection method.

[0056] In this application, the purpose of the second setting of the BERT model is to dynamically select the appropriate fully connected layer and output layer according to the medical field to which the input text belongs, so as to adapt to the needs of different medical fields. Based on this setting, the flexibility of the model can be improved, so that it can better process text data in different medical fields, thereby achieving optimization for specific fields and improving the accuracy of the model.

[0057] BERT model setting three: The BERT model is configured with a loop termination module, which is used to terminate the loop when the loop rounds of the BERT model reach the round threshold, and instruct the dynamic connection module to reset the layer connection according to the parent medical field of the medical field to which the currently identified case to be analyzed belongs.

[0058] In this application, the purpose of setting three of the BERT model is to automatically terminate the training process after the cyclic training of the model reaches a predetermined number of rounds. If satisfactory results are not obtained in the current medical field, it will fall back to the previous medical field and reconfigure the network connection to continue training. Based on this setting, on the one hand, unnecessary training rounds can be avoided and computing resources can be saved. On the other hand, retraining in the previous medical field can help solve the problem of insufficient data in a specific field and improve the generalization ability of the model.

[0059] BERT model setting 4: The BERT model is configured with a startup module, which is used to start the loop when the layer connections are set up.

[0060] In this application, the purpose of BERT model setting four is to start the training or reasoning process of the model after the network layer connection setting is completed. It is a startup node prepared for BERT model setting two and BERT model setting three, so that the model can correctly enter the data processing flow after the internal structure is adjusted, ensuring the complete functionality of the model.

[0061] For example, the present application may perform linear interpolation fusion in the following manner.

[0062] Figure 2 A flow chart of a linear interpolation fusion method is shown, as Figure 2 As shown, the process includes the following steps S201 to S203.

[0063] In step S201, feature vectors of two different cases to be analyzed in the target data set are selected as reference points.

[0064] In step S202, a linear interpolation coefficient is calculated according to the distance between two eigenvectors.

[0065] In step S203, a new feature vector is generated based on the linear interpolation coefficients.

[0066] For example, the present application may expand the target data set in the following manner.

[0067] Figure 3 A flow chart of a method for expanding a target data set is shown, such as Figure 3 As shown, the process includes the following steps S301 to S302.

[0068] In step S301, the image data of the same dimension in the target data set is fused with feature maps by improved linear interpolation, and reconstructed back to the image space through a decoder to generate a new medical image.

[0069] In step S302, the text data in the target data set is expanded based on the new medical image processed by the improved linear interpolation and combined with the pre-trained natural language processing model.

[0070] In one implementation, an improved linear interpolation formula may be used to perform linear interpolation calculation. The improved linear interpolation formula may be, for example, F mixed =α·F A +β·F B .

[0071] Among them, F mixed represents the result after interpolation, F A 、F BThey represent the feature maps of image data of the same dimension, α represents the interpolation parameter, and β represents the nonlinear degree control parameter. When β = 0, the improved linear interpolation degenerates into the standard linear interpolation. When β > 0, the interpolation result will be biased towards F in a nonlinear way according to the value of α. A or F B .

[0072] This application can use the above improved linear interpolation formula for linear interpolation calculation, which allows the model to have greater flexibility in fusing image features and can adjust F according to specific circumstances. A and F B When β>0, the bias of the interpolation result can be controlled by adjusting the value of α, which has higher sensitivity when dealing with complex or specific types of cases. In addition, by introducing nonlinear control parameters, the model can better adapt to the characteristics of different cases, improve the effect of feature fusion and the accuracy of case analysis.

[0073] In another embodiment, an optimized linear interpolation formula may be used to perform linear interpolation calculation. The optimized linear interpolation formula may be, for example, F mixed =α′·F A +β′·F B .

[0074] Among them, F mixed represents the result after interpolation, F A 、F B They represent the feature maps of image data of the same dimension, and α′ and β′ represent the optimized interpolation parameters. Among them, α′ is calculated by the formula We get β′ by the formula Where C represents the complexity factor of the case to be analyzed, T represents the weight of the case type to be analyzed, D represents the time weight, and α and β represent the basic difference parameters respectively.

[0075] Among them, when C = 0 and D = 0, the improved linear interpolation degenerates into the standard linear interpolation. When C≠0 or D≠0, the interpolation result will be biased towards F in a nonlinear way according to the values ​​of C and D. A or F B .

[0076] The present application can use the above optimized linear interpolation formula for linear interpolation calculation, which can automatically adjust the interpolation parameters according to the complexity, type and time factors of the case to be analyzed, so that the interpolation process is more in line with the actual situation. When C≠0 or D≠0, the interpolation result will be biased towards F in a nonlinear manner according to the values ​​of C and D. A or F BThis means that the model can be adjusted according to the specific attributes of the case, improving the relevance and accuracy of the analysis results. By optimizing the interpolation parameters, the model can better handle diverse case data, improve the quality of feature fusion, and thus improve the overall effect of case analysis.

[0077] In the present application, the step of linear interpolation can be performed based on one of the above linear interpolation formulas. Compared with the improved linear interpolation formula, the optimized linear interpolation formula has higher accuracy and higher computational cost, so in actual use, a choice can be made between the two based on the preference of case analysis, such as selecting the improved linear interpolation formula to ensure timeliness, or selecting the optimized linear interpolation formula to ensure analysis accuracy.

[0078] The present application also provides a system embodiment that is based on the above embodiment, which is used to implement the method steps of the above embodiment. The explanation based on the same name meaning is the same as the above embodiment, and has the same technical effect as the above embodiment, which will not be repeated here.

[0079] like Figure 4 As shown, the present application provides a case analysis system 400 based on deep learning, including a data acquisition module 401, an image feature extraction module 402, a text feature analysis module 403, a feature fusion module 404, a data set expansion module 405, and an input and output module 406.

[0080] The data acquisition module 401 is used to acquire a target data set including image data and text data. The image data and text data are respectively data obtained based on the material cases. The image feature extraction module 402 is used to extract features of the image data in the target data set based on the pre-trained convolutional neural network. The text feature analysis module 403 is used to input the extracted feature vector into the pre-trained natural language processing model for analyzing the text data in the target data set. The feature fusion module 404 is used to generate a new feature vector by linear interpolation fusion based on the image feature vector processed by the convolutional neural network and the text feature vector processed by the natural language processing model. The data set expansion module 405 is used to expand the target data set based on the generated new feature vector, and train the deep learning model based on the expanded target data set to obtain the trained deep learning model. The input-output module 406 is used to input the case to be analyzed as input into the trained deep learning model to obtain the case analysis result output by the trained deep learning model.

[0081] In one embodiment, the convolutional neural network is a pre-trained ResNet50 model.

[0082] In one embodiment, the ResNet50 model adopts the following settings: the preprocessing frame of the ResNet50 model is configured with a standardization module, and the data input to the preprocessing frame is processed by the standardization module and output as a pixel value with normalized characteristics, and the value range of the pixel value conforms to 0 to 1. The ResNet50 model is configured with a rotation module, a scaling module, a flipping module, and a randomization module, and the randomization module performs random and repeatable extraction and parallel arrangement in the rotation module, the scaling module, and the flipping module to obtain a module array. The module array is used to output image feature vectors of different generalization degrees based on multiple different output heads matching the number of columns. The ResNet50 model is configured with an architecture fine-tuning module, which is used to increase or decrease the number of convolution layers according to the characteristics of the medical images of the case to be analyzed. Among them, the randomization module performs random and repeatable extraction based on the number of convolution layers and arranges them in parallel so that the number of convolution layers is consistent with the number of rows of the module array.

[0083] In one embodiment, the natural language processing model is a pre-trained BERT model.

[0084] In one embodiment, the BERT model adopts the following settings: the BERT model performs primary pre-training based on general text data and secondary pre-training based on medical field text data. Among them, the medical field text data is configured with a customized word segmenter based on a medical vocabulary. The BERT model is configured with a network layer dynamic connection module, which is used to query the matching fully connected layer and output layer according to the medical field when the medical field to be analyzed is determined based on text recognition, and the queried fully connected layer and output layer are set as layer connections according to the preconfigured connection method. The BERT model is configured with a loop termination module, which is used to terminate the loop when the loop round of the BERT model reaches the round threshold, and instruct the dynamic connection module to reset the layer connection according to the medical field of the medical field to which the currently identified case to be analyzed belongs. The BERT model is configured with a startup module, which is used to start the loop when the layer connection is set.

[0085] In one embodiment, the feature fusion module 404 performs linear interpolation fusion in the following manner: select feature vectors of two different cases to be analyzed in the target data set as reference points, calculate linear interpolation coefficients based on the distance between the two feature vectors, and generate new feature vectors based on the linear interpolation coefficients.

[0086] In one embodiment, the feature fusion module 404 expands the target data set in the following manner: the image data of the same dimension in the target data set is fused with feature maps by improved linear interpolation, and reconstructed back to the image space by a decoder to generate a new medical image. Based on the new medical image processed by improved linear interpolation, combined with a pre-trained natural language processing model, the text data in the target data set is expanded.

[0087] In one implementation, the linear interpolation formula includes: mixed =α·F A +β·F B Among them, F mixed represents the result after interpolation, F A 、F B They represent the feature maps of image data of the same dimension, α represents the interpolation parameter, and β represents the nonlinear degree control parameter. When β=0, the improved linear interpolation degenerates into the standard linear interpolation. When β>0, the interpolation result will be biased towards F in a nonlinear way according to the value of α. A or F B .

[0088] In one implementation, the linear interpolation formula includes: mixed =α′·F A +β′·F B Among them, F mixed represents the result after interpolation, F A 、F B They represent the feature maps of image data of the same dimension, α′ and β′ represent the optimized interpolation parameters, which are obtained using the following formula: Among them, C represents the complexity factor of the case to be analyzed, T represents the weight of the case type to be analyzed, D represents the time weight, and α and β represent the basic difference parameters respectively.

[0089] Regarding the system in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0090] Although operations are described in a particular order in the drawings, this should not be understood as requiring that the operations be performed in the particular order shown or in serial order, or that all shown operations be performed to achieve desired results. In certain circumstances, multitasking and parallel processing may be advantageous.

[0091] The methods and devices of the present application can be implemented using standard programming techniques, using rule-based logic or other logic to implement various method steps. It should also be noted that the words "device" and "module" used herein and in the claims are intended to include implementations based on one or more lines of software code and / or hardware implementations and / or devices for receiving input.

[0092] Any steps, operations or procedures described herein may be performed or implemented based on one or more hardware or software modules, either alone or in combination with other devices. In one embodiment, the software module is implemented based on a computer program product including a computer-readable medium containing computer program code, which can be executed by a computer processor to perform any or all of the described steps, operations or procedures.

[0093] The foregoing description of the implementation of the present application has been given for the purpose of illustration and description. The foregoing description is not exhaustive nor is it intended to limit the present application to the exact form disclosed, and various variations and modifications may exist according to the above teachings or may be obtained from the practice of the present application. These embodiments are selected and described in order to illustrate the principles of the present application and its practical application, so that those skilled in the art can utilize the present application in various embodiments and various modifications suitable for the specific purpose contemplated.

[0094] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0095] It can be further understood that, unless otherwise specified, “connection” includes a direct connection without other components between the two, and also includes an indirect connection with other components between the two.

[0096] It is further understood that, although the operations are described in a specific order in the drawings in the embodiments of the present application, it should not be understood as requiring the operations to be performed in the specific order or serial order shown, or requiring the execution of all the operations shown to obtain the desired results. In certain environments, multitasking and parallel processing may be advantageous.

[0097] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to encompass any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the field of the present application that are not disclosed in the present application. The specification and embodiments are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following scope of rights.

[0098] It should be understood that the present application is not limited to the precise structures described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the scope of the appended claims.

[0099] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A case analysis method based on deep learning, characterized in that: include: Acquire a target data set including image data and text data; wherein the image data and the text data are respectively data obtained based on the material case; Extract features from image data in the target dataset based on a pre-trained convolutional neural network; The extracted feature vectors are input into a pre-trained natural language processing model to analyze the text data in the target dataset; Generate a new feature vector by linear interpolation fusion based on the image feature vector processed by the convolutional neural network and the text feature vector processed by the natural language processing model; Expanding the target data set based on the generated new feature vector, and training the deep learning model based on the expanded target data set to obtain a trained deep learning model; The case to be analyzed is taken as input and input into the trained deep learning model to obtain the case analysis result output by the trained deep learning model.

2. The method according to claim 1, characterized in that The convolutional neural network is a pre-trained ResNet50 model.

3. The method according to claim 1, characterized in that The ResNet50 model uses the following settings: The preprocessing box of the ResNet50 model is configured with a standardization module, and the data input to the preprocessing box is processed by the standardization module and output as a pixel value with normalized characteristics, and the value range of the pixel value conforms to 0 to 1; The ResNet50 model is configured with a rotation module, a scaling module, a flipping module, and a randomization module, wherein the randomization module randomly and repeatedly extracts and arranges in parallel the rotation module, the scaling module, and the flipping module to obtain a module array; The module array is used to output image feature vectors of different generalization degrees based on a plurality of different output heads matching the number of columns; The ResNet50 model is configured with an architecture fine-tuning module for increasing or decreasing the number of convolutional layers according to the characteristics of the medical images of the case to be analyzed; wherein the randomization module performs random and repeatable extraction based on the number of convolutional layers and arranges them in parallel so that the number of convolutional layers is consistent with the number of rows of the module array.

4. The method according to claim 1, characterized in that: The natural language processing model is a pre-trained BERT model.

5. The method according to claim 4, characterized in that The BERT model uses the following settings: The BERT model is pre-trained at the first level based on general text data and pre-trained at the second level based on medical field text data; wherein the medical field text data is configured with a customized word segmenter based on a medical word library; The BERT model is configured with a network layer dynamic connection module, which is used to query the matching fully connected layer and output layer according to the medical field when the medical field to which the case to be analyzed belongs is determined based on text recognition, and set the queried fully connected layer and output layer as layer connections according to the preconfigured connection mode; The BERT model is configured with a loop termination module, which is used to terminate the loop when the loop round of the BERT model reaches the round threshold, and instruct the dynamic connection module to reset the layer connection according to the upper-level medical field of the medical field to which the currently identified case to be analyzed belongs; The BERT model is configured with a startup module for starting a loop when the layer connection is set.

6. The method according to claim 1, characterized in that The linear interpolation fusion includes: Selecting feature vectors of two different cases to be analyzed in the target data set as reference points; Calculating a linear interpolation coefficient according to the distance between the two eigenvectors; Based on the linear interpolation coefficients, a new feature vector is generated.

7. The method according to claim 6, characterized in that The expanding the target data set includes: The image data of the same dimension in the target data set are fused with feature maps by improved linear interpolation, and reconstructed back to image space by a decoder to generate a new medical image; According to the new medical image processed by improved linear interpolation and combined with a pre-trained natural language processing model, the text data in the target data set is expanded.

8. The intelligent case analysis method based on deep learning according to claim 7 is characterized in that: The linear interpolation formula includes: F mixed =α·F A +β·F B ; Among them, F mixed represents the result after interpolation, F A 、F B They respectively represent the feature maps of image data of the same dimension, α represents the interpolation parameter, and β represents the nonlinear degree control parameter.

9. The intelligent case analysis method based on deep learning according to claim 7 is characterized in that: The linear interpolation formula includes: F mixed =α′·F A +β′·F B ; Among them, F mixed represents the result after interpolation, F A 、F B They represent the feature maps of image data of the same dimension, α′ and β′ represent the optimized interpolation parameters, which are obtained using the following formula: Among them, C represents the complexity factor of the case to be analyzed, T represents the weight of the case type to be analyzed, D represents the time weight, and α and β represent the basic difference parameters respectively.

10. A case analysis system based on deep learning, characterized in that: include: A data acquisition module, used to acquire a target data set including image data and text data; wherein the image data and the text data are respectively data obtained based on the material case; An image feature extraction module is used to extract features from image data in a target data set based on a pre-trained convolutional neural network; A text feature analysis module is used to input the extracted feature vector into a pre-trained natural language processing model for analyzing the text data in the target data set; A feature fusion module is used to generate a new feature vector by linear interpolation fusion based on the image feature vector processed by the convolutional neural network and the text feature vector processed by the natural language processing model; A data set expansion module is used to expand the target data set based on the generated new feature vector, and train the deep learning model based on the expanded target data set to obtain a trained deep learning model; The input-output module is used to input the case to be analyzed as input to the trained deep learning model to obtain the case analysis result output by the trained deep learning model.

Citation Information

Patent Citations

  • Skin disease image classification system based on multi-modal data input

    CN111444960A

  • Histopathology image classification method based on color deconvolution and self-attention model

    CN114066874A

  • Coronary heart disease prediction system based on multi-mode carotid artery data

    CN114464326A

  • Multi-modal pre-training stomach tumor classification system

    CN115115897A

  • Audio and video title generation method and device, equipment and storage medium

    CN117688943A