Training method of trauma assessment model, trauma assessment method and electronic equipment
By using medical image segmentation networks and fully convolutional neural networks in FAST ultrasound examination, ultrasound images are processed and network layers are adjusted, and the problem of insufficient detection capabilities of deep and subtle damage is solved, improving the accuracy and efficiency of diagnosis of trauma patients.
Patent Information
- Application Number
- CN202411194047.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-28
- Publication Date
- 2024-12-03
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The existing FAST ultrasound examination has insufficient detection ability in deep and subtle injuries, resulting in low accuracy and efficiency of diagnosis in trauma patients.
Using the trauma assessment model training method, ultrasound images are obtained through medical image segmentation network and fully convolutional neural network, mask image processing is performed to distinguish effusion and non-effusion areas, and the network layer of the encoder is analyzed and the response measurement is performed. Adjust and combine multiple fusion solutions to improve classification performance indicators.
Trauma assessment models that can identify deep and subtle injuries are trained to improve the accuracy and efficiency of diagnosis of trauma patients.
Smart Images

Figure CN119068290A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of medical image processing, and in particular to a trauma assessment model training method, a trauma assessment method and an electronic device. Background Art
[0002] In the medical field, the rapidity and accuracy of trauma assessment are crucial to improving the survival rate of patients. Ultrasound imaging technology, with its convenience, non-invasiveness and radiation-free characteristics, plays an increasingly important role in the rapid diagnosis of trauma patients.
[0003] Focused Assessment with Sonography for Trauma (FAST) is an innovative application of ultrasound technology in trauma assessment. It uses a standardized examination process to quickly assess whether the patient has free fluid in the pericardium, thorax or abdominal cavity. The six key sections of FAST ultrasound examination are: subcostal four-chamber heart section, right intercostal oblique section, right coronary section, left intercostal oblique section, left coronary section and pelvic section.
[0004] However, FAST ultrasound examination is still insufficient in its ability to assess deep injuries and detect subtle injuries, resulting in low accuracy and efficiency in diagnosing trauma patients. Summary of the invention
[0005] In view of this, an object of the embodiments of the present invention is to provide a trauma assessment model training method, a trauma assessment method and an electronic device to at least partially improve the above-mentioned problem.
[0006] In order to achieve the above purpose, the technical solution adopted by the embodiment of the present invention is as follows:
[0007] In a first aspect, an embodiment of the present invention provides a trauma assessment model training method, which is applied to a model training system, wherein the model training system includes a trauma assessment model, wherein the trauma assessment model includes a medical image segmentation network and a fully convolutional neural network, wherein the fully convolutional neural network includes at least two encoders, and wherein the encoder includes multiple network layers. The method includes:
[0008] Acquire multiple groups of ultrasound images, each group of ultrasound images including ultrasound images corresponding to key areas of the trauma patient;
[0009] Inputting each group of the ultrasound images into the medical image segmentation network, segmenting each ultrasound image in each group of ultrasound image samples, and obtaining a mask image corresponding to each ultrasound image for distinguishing between effusion and non-effusion areas;
[0010] Inputting the mask images corresponding to each group of ultrasound images into the corresponding encoders for feature extraction, performing response metric analysis on each network layer of each encoder during the feature extraction process to obtain the importance of each network layer to the classification result, and adjusting the network layer of each encoder based on the importance; wherein the encoders correspond to the mask images one by one;
[0011] According to each of the importances, a plurality of fusion schemes are combined to fuse the output data of the network layers of each of the encoders, and each classification performance index of the trauma assessment model for the group of ultrasound image samples is calculated when the trauma assessment model uses each of the fusion schemes;
[0012] The fusion scheme with the best classification performance index is selected as the fusion scheme of the trauma assessment model to obtain the trained trauma assessment model.
[0013] In a second aspect, an embodiment of the present invention provides a trauma assessment method, characterized in that it is applied to a trauma assessment model, the trauma assessment model includes a medical image segmentation network and a fully convolutional neural network, and the method includes:
[0014] Acquire a group of ultrasonic images to be detected, wherein the group of ultrasonic images to be detected includes ultrasonic images corresponding to key areas of the trauma patient respectively;
[0015] Inputting the ultrasonic image group to be detected into the medical image segmentation network, segmenting each of the ultrasonic images respectively, and obtaining a mask image corresponding to each of the ultrasonic images for distinguishing between effusion and non-effusion areas;
[0016] Each of the mask images is input into the fully convolutional neural network to obtain a classification result of the ultrasonic image group to be detected.
[0017] In a third aspect, an embodiment of the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method described in any one of the first aspect above is implemented; and / or the method described in the second aspect above is implemented.
[0018] Compared with the prior art, the embodiment of the present invention provides a trauma assessment model training method, a trauma assessment method and an electronic device. When training the trauma assessment model, each ultrasound image in each group of ultrasound image samples is segmented using a medical image segmentation network to obtain a mask image corresponding to each ultrasound image that distinguishes between effusion and non-effusion areas, and then each mask image is input into each corresponding encoder of the full convolutional neural network, and a response metric analysis is performed on each network layer of each encoder to obtain the importance of each network layer to the classification result, and the network layer of each encoder is adjusted based on each importance. In addition, according to the importance, multiple fusion schemes are combined to fuse the output data of the network layer of each encoder, and then the classification performance index of the trauma assessment model is calculated when each fusion scheme is used, and the fusion scheme with the best classification performance index is selected as the fusion scheme of the trauma assessment model to obtain the trained trauma assessment model. In this way, a trauma assessment model that can identify deep injuries and subtle injuries in ultrasound images can be trained to improve the accuracy and efficiency of diagnosis of trauma patients.
[0019] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.
[0021] Figure 1 A schematic structural block diagram of an electronic device provided by an embodiment of the present invention;
[0022] Figure 2 A schematic structural block diagram of a model training system provided by an embodiment of the present invention;
[0023] Figure 3 A schematic diagram of a flow chart of a trauma assessment model training method provided by an embodiment of the present invention;
[0024] Figure 4 Another schematic diagram of a flow chart of a trauma assessment model training method provided by an embodiment of the present invention;
[0025] Figure 5 Another schematic diagram of a flow chart of a trauma assessment model training method provided by an embodiment of the present invention;
[0026] Figure 6A schematic structural block diagram of an encoder provided by an embodiment of the present invention;
[0027] Figure 7 A schematic diagram of network layer fusion of an encoder provided by an embodiment of the present invention;
[0028] Figure 8 A schematic diagram of a process flow of a trauma assessment method provided by an embodiment of the present invention;
[0029] Fig. 9 A network architecture diagram of a trauma assessment model provided in an embodiment of the present invention.
[0030] Icons: 100-electronic device; 101-memory; 102-communication interface; 103-processor; 104-bus; 10-model training system; 40-trauma assessment model; 41-medical image segmentation network; 42-fully convolutional neural network; 421-encoder; 4211-network layer; 4212-alternative fusion layer; 422-first encoder; 423-second encoder; 424-third encoder; 427-sixth encoder. DETAILED DESCRIPTION
[0031] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.
[0032] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0033] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings. At the same time, in the description of the present invention, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.
[0034] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0035] In the medical field, the rapidity and accuracy of trauma assessment are crucial to improving the survival rate of patients. Ultrasound imaging technology, with its convenience, non-invasiveness and radiation-free characteristics, plays an increasingly important role in the rapid diagnosis of trauma patients.
[0036] As an innovative application of ultrasound technology in trauma assessment, FAST can quickly assess whether the patient has free fluid in the pericardium, thorax or abdominal cavity through a standardized examination process. The six key sections of FAST ultrasound examination are: subcostal four-chamber heart section, right intercostal oblique section, right coronary section, left intercostal oblique section, left coronary section and pelvic section.
[0037] However, although FAST has shown great potential in trauma assessment, it still faces some challenges. The quality of ultrasound images and the accuracy of diagnosis depend largely on the skills and experience of the operator. In addition, the sensitivity of ultrasound may be limited in some cases, and it needs to be combined with other imaging methods, such as CT scans, to obtain more comprehensive diagnostic information. FAST ultrasound examination is still insufficient in its ability to assess deep injuries and detect subtle injuries, resulting in low accuracy and efficiency in the diagnosis of trauma patients.
[0038] Based on the above situation, the embodiment of the present invention provides a trauma assessment model training method, a trauma assessment method and an electronic device, wherein multiple encoders are set for the trauma assessment model, each encoder corresponds to an ultrasound image of a key area of a trauma patient, and the importance of the classification results of each network layer in each encoder is adjusted based on each importance, and multiple fusion schemes of the trauma assessment model are combined according to each importance, and the classification performance index of the trauma assessment model is calculated for each set of fusion schemes, and the fusion scheme with the best classification performance index is selected as the fusion scheme of the trauma assessment model to obtain the trained trauma assessment model. The trauma assessment model is enabled to identify deep injuries and subtle injuries.
[0039] To implement the process steps and functions of each example of the present invention, please refer to Figure 1 , Figure 1 A schematic structural block diagram of an electronic device provided by an embodiment of the present invention. The electronic device 100 includes a memory 101 and a processor 103, and the memory 101 and the processor 103 are electrically connected to each other directly or indirectly to realize data transmission or interaction. For example, these elements can be electrically connected to each other through one or more communication buses 104 or signal lines. The memory 101 can be used to store software programs and modules, and the processor 103 executes the software programs and modules stored in the memory 101 to perform various functional applications and data processing.
[0040] The electronic device 100 may be, but is not limited to, a personal computer (PC), a server, a distributed computer, etc. It is understandable that the electronic device 100 is not limited to a physical server, but may also be a virtual machine on a physical server, a virtual machine built on a cloud platform, etc., which can provide the same functions as the server or virtual machine. The operating system of the electronic device 100 may be, but is not limited to, a Windows system, a Linux system, etc.
[0041] Among them, the memory 101 can be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), etc.
[0042] The communication connection between the electronic device 100 and an external device is achieved through at least one communication interface 102 (which can be wired or wireless).
[0043] The processor 103 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the embodiment of the present invention can be completed by the hardware integrated logic circuit in the processor 103 or the instructions in the form of software. The processor 103 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.
[0044] Understandably, Figure 1 The structure shown is for illustration only. The electronic device 100 may also include Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown. Figure 1 Each component shown in the figure can be implemented by hardware, software or a combination thereof.
[0045] The following is an exemplary description of the trauma assessment model training method provided by the present invention. Specifically, Figure 2 A schematic structural block diagram of a model training system provided by an embodiment of the present invention, see Figure 2 The trauma assessment model training method is applied to a model training system 10, which includes a trauma assessment model 40, which includes a medical image segmentation network 41 and a fully convolutional neural network 42, and the fully convolutional neural network 42 includes at least two encoders 421, each encoder 421 includes multiple network layers 4211. The model training system 10 can be deployed in the above Figure 1 An electronic device 100 is shown.
[0046] Furthermore, Figure 3 A flowchart of a trauma assessment model training method provided by an embodiment of the present invention, combined with Figure 2 and Figure 3 , the method comprises the following steps:
[0047] S210: Acquire multiple groups of ultrasound images, wherein each group of ultrasound images includes ultrasound images corresponding to each key area of the trauma patient.
[0048] Preferably, each group of ultrasound images may include ultrasound images of six key sections of FAST ultrasound examination, and the six key sections include: subcostal four-chamber view, right intercostal oblique view, right coronary view, left intercostal oblique view, left coronary view and pelvic view.
[0049] S220: Input each group of ultrasound images into a medical image segmentation network, segment each ultrasound image in each group of ultrasound image samples, and obtain a mask image corresponding to each ultrasound image that distinguishes between effusion and non-effusion areas.
[0050] In this embodiment, the medical image segmentation network is a Swin-Unet model. The application of the Swin-Unet model realizes efficient automatic segmentation of ultrasound images of six key sections of trauma patients. The model uses an encoder-decoder architecture to accurately distinguish between effusion and non-effusion areas through feature extraction and upsampling processes. After segmentation by the Swin-Unet model, a mask image corresponding to each ultrasound image can be obtained.
[0051] S230: Inputting each mask image corresponding to each group of ultrasound images into the corresponding encoder for feature extraction, performing response metric analysis on each network layer of each encoder during the feature extraction process, obtaining the importance of each network layer to the classification result, and adjusting the network layer of each encoder based on the importance. The encoder corresponds to the mask image one by one.
[0052] The obtained mask images are respectively input into the corresponding encoders. For example, in a preferred embodiment, each group of ultrasound images includes 6 ultrasound images. After segmentation by Swin-Unet, 6 mask images are obtained. The 6 mask images are respectively input into the corresponding 6 encoders, which can be ResNet50. The 6 encoders perform feature extraction on their respective mask images. During the feature extraction process, response metric analysis is performed on each network layer of the 6 encoders to determine the recognition ability of each network layer of each encoder for the effusion of its respective mask image, and the importance of each network layer to the classification result.
[0053] And according to the importance of each network layer, each network layer is adjusted, which may include adjusting convolution parameters and increasing or decreasing the number of layers to enhance the recognition ability of the features of the mask image of the respective area.
[0054] S240: Combining multiple fusion schemes for fusing the output data of the network layers of each encoder according to the importance, respectively calculating each classification performance index of the trauma assessment model for the group of ultrasound image samples when the trauma assessment model uses each fusion scheme.
[0055] It is necessary to find which network layers of each encoder can perform feature fusion to most effectively combine the effusion information of different key areas, thereby improving the overall segmentation performance.
[0056] The simplest approach is to perform feature fusion on each network layer of the six encoders to evaluate the effects of different fusion layers. However, this method requires too many fusion schemes to be verified, which will result in a large waste of computing and training costs.
[0057] In an embodiment of the present invention, after obtaining the importance of each network layer of each encoder to the classification result, some fusion schemes that may have better classification effects are designed according to the importance, and then the effects of different fusion schemes are evaluated, so as to quickly obtain the optimal fusion scheme.
[0058] S250: Obtaining the fusion scheme with the best performance index of multiple groups of ultrasound selection classification as the fusion scheme of the trauma assessment model, and obtaining the trained trauma assessment model.
[0059] In each training, the network structure of each encoder is adjusted, such as adding or removing network layers, adjusting the convolution kernel size, changing the network depth, etc. The fusion strategy is adjusted, including changing the fused network layer and using different fusion techniques. After training multiple sets of ultrasound images, a mature trauma assessment model is obtained.
[0060] The trauma assessment model training method provided by the embodiment of the present invention sets multiple encoders for the full convolutional neural network of the trauma assessment model, each encoder corresponds to the ultrasound image of the key area of a trauma patient, and adjusts the network layer of each encoder based on the importance of the classification results of each network layer in each encoder, and combines some fusion schemes with better classification effects according to each importance, and then evaluates the classification effects of different fusion schemes, so that the trauma assessment model can be trained quickly. The obtained mature trauma assessment model can identify deep injuries and subtle injuries, and improve the accuracy and efficiency of diagnosis of trauma patients.
[0061] In the above step S230, in order to obtain the importance of each network layer to the classification result, and then adjust the network layer of the encoder, and use it as a reference for the feature fusion of step S240, each network layer of each encoder can be directly evaluated. However, this method removes and retrains each network layer separately, which increases the consumption of computing resources. During the training process, if each network layer is removed once, it may cause too much interference to the basic model and fail to maintain the stability of the model. In a preferred implementation method, the importance of each network layer of each encoder to the classification result can be estimated first, and then some particularly important and unimportant network layers can be screened out based on the estimated importance, and the importance of the classification result of these network layers can be evaluated. For details, see Figure 4 , Figure 4 Another flowchart of a trauma assessment model training method provided by an embodiment of the present invention, the above step S230 may include the following: Figure 4 The following steps are shown:
[0062] S231: For each encoder, by analyzing each network layer of the encoder, an estimated importance of each network layer to the classification result of the trauma assessment model is obtained.
[0063] The importance of each network layer to the classification results of the trauma assessment model can be estimated by analyzing the features of each network layer output or by visual analysis. Among them, feature analysis can be performed by analyzing the feature vectors output by the network layer, etc., and feature selection or feature importance evaluation methods (such as gradient-based methods, model-based methods, etc.) can be used to determine which features contribute most to the classification results. Visualization techniques can include t-SNE, PCA, etc.
[0064] S232: After cyclically removing the network layers that meet the preset importance conditions according to the estimated importances, the importance of each removed network layer to the classification result of the trauma assessment model is calculated.
[0065] After obtaining the estimated importance of each layer, a layer-by-layer ablation experiment can be performed to remove the network layers that meet the preset importance conditions and observe the changes in the importance of the classification results of the trauma assessment model. If the performance drops significantly after removing a certain network layer, it means that the layer has an important impact on the model performance. This method can intuitively reflect the importance of each network layer to the classification results of the trauma assessment model.
[0066] S233: Adjust the network layers of the encoder based on importance.
[0067] According to the importance of each network layer, adjust the architecture of each network layer, such as adding or removing convolutional layers, adjusting the size of convolution kernels, changing the network depth, etc.
[0068] Optionally, the estimated importance may include the contribution and sensitivity to the classification result of the trauma assessment model. Using two dimensions to evaluate the estimated importance can make the comparison between the estimated importance and the real importance more accurate. Specifically, the above step S231 may include the following steps:
[0069] S2331: Input the mask image corresponding to the encoder into the encoder to obtain output data of each network layer of the encoder.
[0070] During the encoder processing of the mask image, the output data of each network layer is recorded for subsequent analysis.
[0071] S2332: Analyze the output data of each network layer to obtain the contribution and sensitivity of each network layer to the classification results of the trauma assessment model.
[0072] Among them, contribution represents the ability of the network layer to capture local details and global context information, and sensitivity represents the response of the network layer to changes in input data.
[0073] Optionally, the contribution can be evaluated through Grad-CAM, which generates a corresponding activation map from the output data of each network layer, and estimates the contribution of each network layer to the classification result of the trauma assessment model based on each activation map.
[0074] First, for the input mask image, the encoder is used for forward propagation until the last network layer, and the activation map of each network layer is generated by weighted sum.
[0075] The contribution of each network layer to the classification result can be evaluated by comparing the activation maps generated by different network layers. Generally, if the highlighted area of the activation map of a certain network layer highly overlaps with the actual effusion area, it can be considered that the network layer has a greater contribution to the classification result.
[0076] Sensitivity can be weakly perturbed for the input data of each network layer, and gradient analysis can be performed to obtain the sensitivity of each network layer to the classification results of the trauma assessment model. Among them, weak perturbation can include adding noise, rotating, and scaling the input data for data enhancement. The steps may include:
[0077] 1. Determine how to weakly perturb the input data, such as adding noise, rotating, or scaling.
[0078] 2. Apply the selected perturbation to the input data of each network layer.
[0079] 3. Propagate the perturbed input data forward through the network to obtain the classification result.
[0080] 4. Calculate the gradient of the classification result with respect to the perturbed input data and evaluate the sensitivity.
[0081] 5. According to the size of the gradient, analyze the sensitivity of each network layer to the classification results.
[0082] High gradient values indicate that each network layer is more sensitive to the classification results.
[0083] S2333: Determine the estimated importance of each network layer to the classification results of the trauma assessment model based on contribution and sensitivity.
[0084] After obtaining the contribution and sensitivity, the estimated importance can be obtained by directly adding the contribution and sensitivity together, or by weighting them and adding them together. Each contribution and sensitivity can also be sorted separately, and the estimated importance can be calculated based on the sorted values. For example, if the sorted values of the two are directly added together, and the contribution ranking of a network layer is 2 and the sensitivity ranking is 3, then its estimated importance is 5.
[0085] In the above step S232, there are multiple ways to calculate the importance of the network layer to the classification result of the trauma assessment model based on each estimated importance. In one possible implementation, the importance of each network layer can be evaluated through an ablation experiment, and the changes in classification performance can be observed by removing the network layers layer by layer. If the performance drops significantly after removing a certain network layer, it means that the network layer has an important impact on the performance of the trauma assessment model. Specifically, the above step S232 may include the following steps:
[0086] S2321: Calculate a first confusion matrix of the trauma assessment model, and calculate a first metric of the trauma assessment model based on each first confusion matrix.
[0087] First, the first metric is calculated without any changes to the trauma assessment model. The first metric may include recall, precision, and F1 score. The first metric is calculated first as a reference for comparison with subsequent changes to the encoder.
[0088]
[0089]
[0090] Among them, Accuracy is accuracy, Precision is precision, Recall is recall, F1Score is F1 score, TP is true positive, FP is false positive, TN is true negative, and FN is false negative.
[0091] S2322: Remove in sequence each network layer whose estimated importance is less than the first preset importance or greater than the second importance.
[0092] The network layers that meet certain preset conditions are removed in sequence. In this embodiment, the condition is that the estimated importance is less than the first preset importance or greater than the second preset importance, wherein the first preset importance is less than the second preset importance. The network layer whose estimated importance is less than the first preset importance indicates that the network layer may not be important to the final classification result, and the network layer whose estimated importance is greater than the second preset importance indicates that the network layer may be particularly important to the final classification result. These two types of network layers are evaluated again.
[0093] S2323: Calculate a second confusion matrix of the trauma assessment model, and calculate a second metric of the trauma assessment model based on each second confusion matrix.
[0094] After removing the network layer that meets the above conditions, the second metric of the trauma assessment model is calculated according to the same calculation method in step S2321.
[0095] S2324: According to the first metric and each of the second metric, the importance of the network layer corresponding to each of the second metric to the classification result of the trauma assessment model is obtained.
[0096] Each of the second metric indicators is marked with the first metric indicator respectively, and the importance of each network layer to the classification result of the trauma assessment model is determined according to the size of the difference between the two.
[0097] For example, the first preset importance is 2, and the second preset importance is 8. When the estimated importance of the network layer is less than 2, it means that the network layer may not be very important, and then the actual importance of the network layer is calculated; if the estimated importance of the network layer is greater than 8, it means that the network layer may be very important, and then the actual importance of the network layer is calculated. Finally, adjustments are made according to the actual importance of the network layer.
[0098] In the above step S233, after calculating the importance of each network layer to the classification result of the trauma assessment model, it is necessary to adjust the network layer of the encoder based on each importance, and the network layer may be added or removed, the convolution kernel size may be adjusted, the network depth may be changed, etc. Optionally, the above step S233 may include:
[0099] 1. Delete the network layers whose importance is less than the third preset importance.
[0100] The third preset importance can be the same as the first preset importance, that is, the third preset importance is 2. When the calculated importance is still less than 2, the network layer is deleted. Preferably, the third preset importance is less than the first preset importance, such as the first preset importance is 3 and the third preset importance is 2. The purpose of this is to expand the base number of the estimated selected network layers. When calculating the importance, it is possible to find a network layer with a real importance of less than 2 among the network layers with an estimated importance between 2 and 3.
[0101] 2. For network layers whose importance is less than the fourth preset importance, the number of channels is reduced.
[0102] For network layers with importance less than the fourth preset importance, such as the fourth preset importance is 4, it means that the network layers with importance between 2 and 4 are not very important, but are slightly more important than the network layers with importance less than 2. Therefore, for such network layers, instead of removing them directly, some parameter adjustments can be made, such as reducing the number of channels of the network layer.
[0103] 3. For network layers whose importance is greater than or equal to the fourth preset importance, the number of channels is increased.
[0104] For network layers whose importance is greater than or equal to the fourth preset importance and less than the fifth preset importance, if they are important for the classification result of the trauma assessment model, the parameters are adjusted and the number of channels is increased to further improve their feature capture capabilities.
[0105] 4. Network layers whose importance is greater than the fifth preset importance remain unchanged.
[0106] For network layers with an importance greater than the fifth preset importance, their contribution to the classification results of the trauma assessment model is very large. Therefore, in this case, we often do not make changes to them.
[0107] After analyzing and adjusting each network layer, each encoder has a stronger feature recognition ability for the mask image corresponding to the key area of the corresponding trauma patient, and can better capture the unique effusion characteristics of each key area.
[0108] After adjusting each network layer of each encoder, in order to effectively combine the effusion information of different key areas of trauma patients to obtain comprehensive features and thus perform comprehensive trauma diagnosis, the output features of each network layer of each encoder can also be fused. In a possible implementation method, see Figure 5 , Figure 5 Another flowchart of a trauma assessment model training method provided by an embodiment of the present invention, the above step S240 may include the following: Figure 5 The following steps are shown:
[0109] S241: According to the importance, a preset number of the network layers are selected as candidate fusion layers, and the candidate fusion layers are combined into multiple groups of fusion layer solutions by using a permutation and combination method.
[0110] After calculating the importance of each network layer of each encoder, see Figure 6 , Figure 6A schematic structural block diagram of an encoder provided in an embodiment of the present invention, wherein a network layer of certain data in each encoder is selected as an alternative fusion layer 4212, for example, each encoder 421 selects multiple most important alternative fusion layers 4212 (only two are drawn in the figure), and based on these alternative fusion layers 4212, multiple groups of fusion layer schemes are combined, for example, one-layer fusion, multiple-layer fusion, and other groups of fusion layer schemes.
[0111] S242: Arrange and combine multiple fusion solutions according to the solutions of each fusion layer and the preset fusion strategy.
[0112] The preset fusion strategies include element-level summation, feature map stacking, and weighted summation; multiple fusion layer solutions are combined with the preset fusion strategies to obtain more fusion solutions. For example, this example includes 3 preset fusion strategies, which are recombined into a 3-fold fusion solution.
[0113] S243: using each set of fusion schemes to fuse the output data of each candidate fusion layer, and calculating each classification performance index of the trauma assessment model for the group of ultrasound image samples when the trauma assessment model uses each set of fusion schemes.
[0114] For each set of fusion schemes, the output data of each candidate fusion layer is fused using the set of fusion schemes. After fusion using the fusion schemes, the classification performance index of the injury assessment model is calculated. The classification performance index can refer to the first metric index of the above step S2321.
[0115] Furthermore, in this embodiment, the full convolutional neural network includes six encoders. In step S243, the step of fusing the output data of each candidate fusion layer using each set of fusion schemes may include the following steps:
[0116] S2431: For each fusion scheme, obtain the feature map output by the alternative fusion layer of each encoder.
[0117] S2432: Arrange the encoders in sequence and encode them into a first encoder, a second encoder, a third encoder, a fourth encoder, a fifth encoder, and a sixth encoder respectively.
[0118] S2433: Fusing the feature maps output by each candidate fusion layer of the first encoder with the feature maps output by each candidate fusion layer of the second encoder.
[0119] S2434: Fusing the fused feature map with the feature maps output by the candidate fusion layers of the third encoder, until the fusion of the feature maps output by the candidate fusion layers of the sixth encoder is completed.
[0120] See also Figure 7 , Figure 7 A schematic diagram of the network layer fusion of an encoder provided in an embodiment of the present invention, in this embodiment, includes six encoders 421, a feature map output by an alternative fusion layer of the first encoder 422 is spliced with the output feature map of the network layer of the same level of the second encoder 423 to obtain a first spliced feature, the first spliced feature is input into the next network layer of the second encoder 423, and the first spliced feature is spliced with the output feature map of the third encoder 424 to obtain a second spliced feature, the second spliced feature is input into the next network layer of the third encoder 424, and the second spliced feature is spliced with the output feature map of the fourth encoder, and the fusion is continued in sequence until the fusion of the feature maps output by the alternative fusion layers of the sixth encoder 427 is completed.
[0121] The fusion process can adopt element-level summation, feature map stacking and weighted summation, among which the metadata summation includes: for the element at each position, the corresponding element values in all feature maps are added, and the ReLU activation function is applied to introduce nonlinearity, while ensuring that negative values are set to zero and retaining the most important activation signals.
[0122] In the embodiment of the present invention, the fusion strategy is implemented after the ReLU activation function to ensure that important feature information is retained during the fusion process.
[0123] After completing the above trauma assessment model training, a mature trauma assessment model is obtained. Figure 2 Specifically, Figure 8 A flow chart of a trauma assessment method provided by an embodiment of the present invention is shown in FIG. Figure 8 The trauma assessment method is applied to the mature trauma assessment model, and the trauma assessment method includes: Figure 8 The following steps are shown:
[0124] S310: Acquire a group of ultrasound images to be detected.
[0125] The ultrasonic image group to be detected includes ultrasonic images corresponding to each key area of the trauma patient.
[0126] In this embodiment, six key areas are included, namely, the subcostal four-chamber heart section, the right intercostal oblique section, the right coronary section, the left intercostal oblique section, the left coronary section and the pelvic section.
[0127] S320: Input the ultrasound image group to be detected into the medical image segmentation network, segment each ultrasound image separately, and obtain a mask image corresponding to each ultrasound image that distinguishes between effusion and non-effusion areas.
[0128] S330: Input each mask image into a fully convolutional neural network to obtain a classification result of the ultrasonic image group to be detected.
[0129] For details, see Fig. 9 , Fig. 9 A network architecture diagram of a trauma assessment model provided in an embodiment of the present invention. First, a FAST ultrasound examination is performed on a trauma patient to obtain ultrasound images of six key sections. The six key sections are segmented using Swin-Unet to obtain six mask images that distinguish between effusion and non-effusion areas. For the segmentation method of Swin-Unet, please continue to refer to Fig. 9 . Next, the six mask images are respectively assigned to the corresponding encoders. Each encoder extracts features from the mask image and fuses the features at the key layer. The features output by each encoder are then input into the upsampling layer to increase the spatial dimension of the data, which is used to restore the features from low resolution to high resolution. The features of each encoder are then summed, which can be an element-level summation operation, to merge the feature maps. Finally, the Softmax activation function is used to convert the network output into a category probability, for example, what are the probabilities of the trauma patient having pneumothorax, hemothorax, pericardial effusion, and solid organ injury.
[0130] Through this trauma assessment method, the trauma assessment model can perform comprehensive trauma diagnosis by using the comprehensive features obtained from the multi-regional effusion feature extraction and fusion steps. This not only includes common pneumothorax, hemothorax, pericardial effusion and solid organ injury, but also can identify more complex trauma situations.
[0131] Furthermore, the trauma assessment method may also include:
[0132] The effusion areas of each mask image are marked with different colors to quickly identify and distinguish different types of effusions.
[0133] Through a standardized color coding scheme, different types of effusions or diseases are assigned specific colors, allowing for quick identification and differentiation. This color coding not only improves diagnostic efficiency, but also provides a consistent visual standard for clinical communication and case studies.
[0134] The segmented effusion areas are displayed in different colors through visualization technology, providing an intuitive display method for the effusion conditions in the six key sections. For example: Subcostal four-chamber view: pericardial effusion is indicated in red, because pericardial effusion is usually an emergency and requires immediate attention; right intercostal oblique view: perihepatic effusion is indicated in yellow to distinguish it from other types of effusion; right coronal view: if right pleural effusion is detected, it is displayed in green to highlight abnormal fluid accumulation in the pleural cavity; left intercostal oblique view: perisplenic effusion is indicated in blue to distinguish it from effusion in other areas; left coronary view: left pleural effusion is indicated in purple for quick identification in the visualization display; pelvic view: free fluid in the pelvic cavity, such as hematoma or other fluid accumulation, is indicated in orange to mark the special focus of this area.
[0135] This color-coding scheme not only helps physicians quickly identify and differentiate between different types of effusions, but also provides an intuitive way to interpret and communicate ultrasound images. With this standardized visualization approach, clinicians can more effectively assess the condition of trauma patients and make appropriate treatment decisions.
[0136] This trauma assessment method can provide comprehensive trauma assessment and treatment decision support:
[0137] 1. Comprehensive trauma diagnosis
[0138] Using the comprehensive features obtained from the multi-regional effusion feature extraction and fusion steps, the trauma assessment model is able to perform comprehensive trauma diagnosis. This not only includes common pneumothorax, hemothorax, pericardial effusion and solid organ injury, but also can identify more complex trauma situations. The network's diagnostic ability benefits from the powerful feature learning ability of the deep learning model, which can identify the subtle differences in trauma from the fused features, thereby providing more accurate diagnostic results.
[0139] 2. Treatment decision support
[0140] Based on the diagnosis, the system further provides treatment decision support. Depending on the severity and type of trauma, the system will recommend appropriate treatment measures. For example, for pericardial effusion, the system may recommend pericardiocentesis to relieve cardiac compression; for hemothorax, the system may recommend chest drainage to remove accumulated blood. These recommendations are based on clinical guidelines and a large amount of medical data, aiming to provide doctors with scientific treatment options.
[0141] 3. Optimize clinical treatment process
[0142] Through automated diagnosis and treatment recommendations, our system can significantly optimize the clinical treatment process. It reduces doctors' time consumption in the diagnosis and decision-making process, allowing them to focus more on patient treatment and care. In addition, the treatment recommendations provided by the system take into account the individual differences and specific circumstances of the patients, which helps to achieve personalized treatment.
[0143] 4. Dynamic treatment plan adjustment
[0144] The system can dynamically adjust according to the patient's real-time feedback and treatment effects. This means that the treatment plan is not static, but can be optimized according to the patient's recovery and changes in vital signs, ensuring that the patient always receives the most appropriate treatment.
[0145] In summary, the embodiment of the present invention provides a trauma assessment model training method, a trauma assessment method and an electronic device. When training the trauma assessment model, each ultrasound image in each group of ultrasound image samples is segmented using a medical image segmentation network to obtain a mask image corresponding to each ultrasound image that distinguishes between effusion and non-effusion areas, and then each mask image is input into each corresponding encoder of the full convolutional neural network, and the response metric analysis is performed on each network layer of each encoder to obtain the importance of each network layer to the classification result, and the network layer of each encoder is adjusted based on each importance. In addition, according to the importance, multiple fusion schemes are combined to fuse the output data of the network layer of each encoder, and then the classification performance index of the trauma assessment model is calculated when each fusion scheme is used, and the fusion scheme with the best classification performance index is selected as the fusion scheme of the trauma assessment model to obtain the trained trauma assessment model. In this way, a trauma assessment model that can identify deep injuries and subtle injuries in ultrasound images can be trained to improve the accuracy and efficiency of diagnosis of trauma patients.
[0146] In the embodiments provided by the present invention, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and a module, a program segment or a part of the code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.
[0147] In addition, the functional modules in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.
[0148] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
[0149] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.
Claims
1. A trauma assessment model training method, characterized in that: Applied to a model training system, the model training system includes a trauma assessment model, the trauma assessment model includes a medical image segmentation network and a fully convolutional neural network, the fully convolutional neural network includes at least two encoders, the encoder includes multiple network layers, and the method includes: Acquire multiple groups of ultrasound images, each group of ultrasound images including ultrasound images corresponding to key areas of the trauma patient; Inputting each group of the ultrasound images into the medical image segmentation network, segmenting each ultrasound image in each group of ultrasound image samples, and obtaining a mask image corresponding to each ultrasound image for distinguishing between effusion and non-effusion areas; Inputting the mask images corresponding to each group of ultrasound images into the corresponding encoders for feature extraction, performing response metric analysis on each network layer of each encoder during the feature extraction process to obtain the importance of each network layer to the classification result, and adjusting the network layer of each encoder based on the importance; wherein the encoders correspond to the mask images one by one; According to each of the importances, a plurality of fusion schemes are combined to fuse the output data of the network layers of each of the encoders, and each classification performance index of the trauma assessment model for the group of ultrasound image samples is calculated when the trauma assessment model uses each of the fusion schemes; The fusion scheme with the best classification performance index is selected as the fusion scheme of the trauma assessment model to obtain the trained trauma assessment model.
2. The method according to claim 1, characterized in that The mask images corresponding to each group of ultrasound images are input into the corresponding encoders for feature extraction. During the feature extraction process, response metric analysis is performed on each network layer of each encoder to obtain the importance of each network layer to the classification result, and the network layer of each encoder is adjusted based on the importance, including: For each of the encoders, by analyzing each of the network layers of the encoders, an estimated importance of each of the network layers to the classification result of the trauma assessment model is obtained; After cyclically removing the network layers that meet the preset importance conditions according to the estimated importances, the importance of each removed network layer to the classification result of the trauma assessment model is calculated; A network layer of the encoder is adjusted based on the importance.
3. The method according to claim 2, characterized in that For each of the encoders, by analyzing each of the network layers of the encoder, obtaining an estimated importance of each of the network layers to the classification result of the trauma assessment model, including: Inputting the mask image corresponding to the encoder into the encoder to obtain output data of each network layer of the encoder; Analyze the output data of each network layer to obtain the contribution and sensitivity of each network layer to the classification result of the trauma assessment model; the contribution represents the ability of the network layer to capture local details and global context information, and the sensitivity represents the response of the network layer to changes in input data; According to the contribution and the sensitivity, the estimated importance of each of the network layers to the classification result of the trauma assessment model is determined.
4. The method according to claim 3, characterized in that The step of analyzing the output data of each network layer to obtain the contribution and sensitivity of each network layer to the classification result of the trauma assessment model includes: By using Grad-CAM, the output data of each network layer is used to generate a corresponding activation map, and based on each activation map, the contribution of each network layer to the classification result of the trauma assessment model is estimated; Weak perturbations are performed on the input data of each network layer, and gradient analysis is performed to obtain the sensitivity of each network layer to the classification result of the trauma assessment model.
5. The method according to claim 2, characterized in that: After cyclically removing the network layers that meet the preset importance conditions according to the estimated importances, the importance of each removed network layer to the classification result of the trauma assessment model is calculated, including: Calculating a first confusion matrix of the trauma assessment model, and calculating a first metric of the trauma assessment model based on each of the first confusion matrices; Sequentially remove each network layer whose estimated importance is less than the first preset importance or greater than the second preset importance; Calculating a second confusion matrix of the trauma assessment model, and calculating a second metric of the trauma assessment model according to each of the second confusion matrices; According to the first metric and each of the second metric, the importance of the network layer corresponding to each of the second metric to the classification result of the trauma assessment model is obtained.
6. The method according to claim 2, characterized in that The step of adjusting a network layer of the encoder based on the importance comprises: Deleting the network layer whose importance is less than the third preset importance; For the network layer whose importance is less than the fourth preset importance, a process of reducing the number of channels is performed; Performing a process of increasing the number of channels on the network layer whose importance is greater than or equal to a fourth preset importance; The network layer whose importance is greater than the fifth preset importance remains unchanged.
7. The method according to claim 1, characterized in that According to each of the importances, a plurality of fusion schemes are combined to fuse the output data of the network layer of each of the encoders, and each classification performance index of the trauma assessment model for the group of ultrasound image samples is calculated when the trauma assessment model uses each of the fusion schemes, including: According to the importance of each, a preset number of the network layers are selected as candidate fusion layers, and the candidate fusion layers are combined into multiple groups of fusion layer solutions by using a permutation and combination method; According to the fusion layer schemes and the preset fusion strategies, multiple fusion schemes are arranged and combined; the preset fusion strategies include element-level summation, feature map stacking and weighted summation; The output data of each candidate fusion layer are fused using each set of the fusion schemes, and each classification performance index of the trauma assessment model for the group of ultrasound image samples is calculated when the trauma assessment model uses each set of the fusion schemes.
8. The method according to claim 7, characterized in that The fully convolutional neural network includes six encoders, and the output data of each candidate fusion layer is fused using each set of the fusion schemes, including: For each set of the fusion schemes, obtaining a feature map output by the candidate fusion layer of each encoder; Arrange the encoders in sequence and encode them into a first encoder, a second encoder, a third encoder, a fourth encoder, a fifth encoder, and a sixth encoder respectively; Correspondingly fusing the feature maps output by each of the candidate fusion layers of the first encoder with the feature maps output by each of the candidate fusion layers of the second encoder; The fused feature map is correspondingly fused with the feature maps output by each of the candidate fusion layers of the third encoder until the fusion of the feature maps output by each of the candidate fusion layers of the sixth encoder is completed.
9. A trauma assessment method, characterized in that: A trauma assessment model trained by the method according to any one of claims 1 to 8, wherein the trauma assessment model comprises a medical image segmentation network and a fully convolutional neural network, and the method comprises: Acquire a group of ultrasonic images to be detected, wherein the group of ultrasonic images to be detected includes ultrasonic images corresponding to key areas of the trauma patient respectively; Inputting the ultrasonic image group to be detected into the medical image segmentation network, segmenting each of the ultrasonic images respectively, and obtaining a mask image corresponding to each of the ultrasonic images for distinguishing between effusion and non-effusion areas; Each of the mask images is input into the fully convolutional neural network to obtain a classification result of the ultrasonic image group to be detected.
10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 8 and / or claim 9 is implemented.
Citation Information
Cited By
Quality control method and system for trauma ultrasound key evaluation image and video
CN119810026A
Knowledge mining method and system for trauma nursing medical field
CN120123398A
