A rib fracture time prediction method and system based on multi-instance learning

Abstract

The application relates to the technical field of image recognition, and discloses a rib fracture time prediction method and system based on multi-instance learning, which comprises the following steps: acquiring CT image data of rib fracture, pre-processing the CT image data to obtain multi-instance slice images; inputting the multi-instance slice images into a multi-scale feature extraction network to extract multi-instance features of four different scales respectively; performing feature alignment processing on the multi-instance features of the first three scales to obtain aligned multi-instance features, generating instance evidence values based on the multi-instance features of the fourth scale, and calculating the contribution weights of the instances according to the instance evidence values; performing feature fusion on the aligned multi-instance features and the multi-instance features of the fourth scale according to the contribution weights to obtain package features of four different scales; and inputting the package features of the four different scales after splicing into a package regressor to obtain a prediction result of the rib fracture time. The rib fracture time can be accurately predicted without relying on a fixed number of images.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image recognition, in particular to a rib fracture time prediction method and system based on multi-instance learning. BACKGROUND

[0002] With the development of forensic identification technology and the refinement of judicial practice needs, the accurate inference of rib fracture injury time has become an important technical challenge in the field of forensic medicine. Rib fracture, as a common injury type in forensic identification, is directly related to the accurate judgment of the formation time, which is related to the causality determination between fracture and trauma and the qualitative analysis of the case. In particular, in the case of old fracture at the fracture site and relatively concealed fracture, accurate time prediction is particularly crucial. Chest computed tomography (CT) can clearly show key anatomical features such as discontinuity of bone cortex, fracture line direction and fracture end displacement due to its high spatial resolution and three-dimensional reconstruction technology advantages, and has become the gold standard for trauma assessment.

[0003] However, the existing technology still has deficiencies in rib fracture time prediction. First, in the identification of human body injury degree, due to the urgent time limit for handling cases, the rib fracture area is smaller in volume and the boundary is blurred compared to the whole CT image. Artificial screening is time-consuming and laborious, and subtle fractures are easy to be missed. Current research on rib fracture injury time based on imageomics technology is less, and existing methods mostly rely on CT images for local labeling and feature extraction, but most of them are limited to rough binary classification discrimination, which is difficult to meet the demand of accurate time prediction.

[0004] Secondly, existing research uses multi-scale and cyclic convolution networks to classify and predict the fracture category and time, but the time interval is usually set to more than 15 days, with a large span and insufficient accuracy. More importantly, due to the need for a fixed number of CT two-dimensional slice images as input for each fracture, data resampling is required, and due to the difference in fracture injury degree and acquisition equipment, noise is inevitably introduced during the resampling process, affecting the accuracy of time prediction.

[0005] In addition, although the multi-instance feature fusion method can reduce the dependence on the fixed number of slice images, existing methods such as average pooling, maximum pooling and gate attention mechanism-based methods are mainly for classification tasks, and the core assumption is that only a few key instances within the bag play a dominant role in the label. However, in the regression task facing continuous labels, all instances tend to contribute to the prediction result to varying degrees, and the contribution degree is greatly affected by feature saliency and image clarity. Therefore, there is a significant difference between the design assumption of the multi-instance method in the classification task and the actual situation in time prediction. SUMMARY

[0006] In view of the above problems, the present application is proposed.

[0007] To solve the above technical problems, the present application provides the following technical solutions: a rib fracture time prediction method based on multi-instance learning, comprising:

[0008] Obtaining CT image data of rib fracture, pre-processing the CT image data to obtain a multi-instance slice image;

[0009] Inputting the multi-instance slice image into a multi-scale feature extraction network to extract four multi-instance features of different scales respectively;

[0010] Performing feature alignment processing on the multi-instance features of the first three scales to obtain aligned multi-instance features, generating instance evidence values based on the multi-instance features of the fourth scale, and calculating the contribution weight of the instance according to the instance evidence values;

[0011] According to the contribution weight, the aligned multi-instance features and the multi-instance features of the fourth scale are respectively fused to obtain four package features of different scales;

[0012] After splicing the four package features of different scales, inputting them into a package regressor to obtain the prediction result of the rib fracture time.

[0013] As a preferred scheme of the rib fracture time prediction method based on multi-instance learning, wherein: the pre-processing of the CT image data to obtain a multi-instance slice image comprises marking a rib region in the CT image data, determining a minimum circumscribed rectangle containing the rib region and performing cropping;

[0014] Performing binaryzation processing on the cropped image to obtain a binary image, calculating the moments of each order according to the binary image and determining the principal axis angle;

[0015] According to the principal axis angle, performing rotation correction and scaling to a uniform size on the cropped image to obtain a multi-instance slice image.

[0016] As a preferred scheme of the rib fracture time prediction method based on multi-instance learning, wherein: inputting the multi-instance slice image into a multi-scale feature extraction network to extract four multi-instance features of different scales respectively comprises performing preliminary feature extraction and size compression on the multi-instance slice image through a stem layer, wherein the stem layer comprises a convolution layer and a down-sampling processing;

[0017] The image output by the stem layer is sequentially input into a first feature extraction stage, a second feature extraction stage, a third feature extraction stage, and a fourth feature extraction stage, each feature extraction stage processing the image output by the stem layer through a pooling attention module to output multi-instance features of a corresponding scale, thereby obtaining four multi-instance features of different scales, wherein the pooling attention module comprises channel attention and spatial attention.

[0018] As a preferred scheme of the rib fracture time prediction method based on multi-instance learning, the feature alignment processing of the multi-instance features of the first three scales to obtain the aligned multi-instance features comprises inputting the multi-instance features of the first three scales into a feature alignment module respectively.

[0019] The feature offset learning and alignment based on the offset of the multi-instance features of the first three scales are performed through deformable convolution.

[0020] The features after deformable convolution are sequentially subjected to layer normalization and linear transformation processing to obtain the aligned multi-instance features.

[0021] As a preferred scheme of the rib fracture time prediction method based on multi-instance learning, the generation of instance evidence values based on the multi-instance features of the fourth scale and the calculation of the contribution weight of the instance according to the instance evidence values comprise inputting the multi-instance features of the fourth scale into a multi-layer perception machine to output distribution parameters.

[0022] Based on evidence theory, the evidence value corresponding to each instance is calculated according to the distribution parameters.

[0023] The distribution parameters are explicitly constrained through an evidence loss function.

[0024] The evidence value corresponding to each instance is normalized to obtain the contribution weight of each instance.

[0025] As a preferred scheme of the rib fracture time prediction method based on multi-instance learning, the feature fusion of the aligned multi-instance features and the multi-instance features of the fourth scale according to the contribution weight to obtain four package features of different scales comprises sorting the contribution weights according to their sizes, determining the instance with the highest evidence value as the main instance, and the remaining instances as reference instances.

[0026] The main instance feature and the reference instance feature are deeply interacted and integrated through an instance feature interaction module.

[0027] The main instance feature and the reference instance feature after the interaction and integration are weighted and accumulated according to the contribution weight.

[0028] The contribution weight is sorted by size from the step of sorting the contribution weight by size to the step of weighted accumulation on the aligned multi-instance feature and the multi-instance feature of the fourth scale respectively to obtain the bag feature of four different scales.

[0029] As a preferred scheme of the rib fracture time prediction method based on multi-instance learning, wherein the step of inputting the bag feature of four different scales into the bag regressor to obtain the prediction result of rib fracture time comprises the step of performing a splicing operation on the bag feature of four different scales in the feature dimension to form the final fusion feature.

[0030] The final fusion feature is input into the bag regressor for processing, wherein the bag regressor is composed of a multi-layer perception machine and an activation function, and outputs the fracture time prediction value in a preset interval range.

[0031] The fracture time prediction value is converted into the actual fracture days to obtain the prediction result of rib fracture time.

[0032] A rib fracture time prediction system based on multi-instance learning, wherein:

[0033] An image preprocessing module acquires CT image data of rib fracture, pre-processes the CT image data, and obtains multi-instance slice images.

[0034] A feature extraction module inputs the multi-instance slice images into a multi-scale feature extraction network to extract multi-instance features of four different scales respectively.

[0035] A feature alignment module performs feature alignment processing on the multi-instance features of the first three scales to obtain aligned multi-instance features, generates instance evidence values based on the multi-instance features of the fourth scale, and calculates the contribution weight of the instance according to the instance evidence values.

[0036] A bag fusion module performs feature fusion on the aligned multi-instance features and the multi-instance features of the fourth scale respectively according to the contribution weight to obtain bag features of four different scales.

[0037] A time prediction module inputs the bag features of four different scales into a bag regressor after splicing to obtain the prediction result of rib fracture time.

[0038] A computer readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the steps of the method of any one of the present application.

[0039] The application provides a rib fracture time prediction method based on multi-instance learning. BRIEF DESCRIPTION OF DRAWINGS

[0040] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0041] Figure 1 The overall flowchart of a rib fracture time prediction method based on multi-instance learning provided by the first embodiment of the present application.

[0042] Figure 2 The data preprocessing flowchart provided by the first embodiment of the present application.

[0043] Figure 3 The structure diagram of the overall network of the present application provided by the first embodiment of the present application.

[0044] Figure 4 The structure diagram of the pooling attention module provided by the first embodiment of the present application.

[0045] Figure 5 The structure diagram of the feature alignment module provided by the first embodiment of the present application.

[0046] Figure 6A structure diagram of a package fusion module provided for a first embodiment of the application.

[0047] Figure 7 A structure diagram of an example feature interaction module provided for the first embodiment of the application. DETAILED DESCRIPTION

[0048] In order to make the above objectives, characteristics and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should belong to the protection scope of the present application.

[0049] Embodiment 1, refer to Figures 1-7 For an embodiment of the present application, a rib fracture time prediction method based on multi-instance learning is provided, comprising:

[0050] S1: Obtain CT image data of rib fracture, and pre-process the CT image data to obtain a multi-instance slice image.

[0051] In this embodiment, as Figure 2 The data pre-processing flowchart is shown in the figure. The process of pre-processing the CT image data includes rib region annotation and cropping, binarization processing and image matrix calculation, and rotation correction and size standardization, aiming to eliminate irrelevant information interference in the image, unify the spatial direction and size specification of the image, and provide standardized input data for subsequent multi-instance feature extraction.

[0052] Specifically, the rib region is annotated in the CT image data, and the minimum circumscribed rectangle containing the rib region is determined and cropped, including manually annotating the rib region of interest in the original CT image, wherein a quadrilateral containing the entire rib region is used for annotation. Considering the irregularity of the manually annotated region, in order to adapt to network input, the minimum circumscribed rectangle containing the annotated region and parallel to the image coordinates on both sides is found, the corresponding region is cropped, and the pixel values outside the annotated region are set to 0.

[0053] By using the cropping strategy of the minimum circumscribed rectangle, a large amount of irrelevant background information in the CT image can be effectively removed, the computing resources are concentrated in the key area of rib fracture, and the efficiency and accuracy of subsequent feature extraction are improved. At the same time, by setting the pixel values outside the annotated region to 0, the contrast between the region of interest and the background is further enhanced, which is conducive to the network focusing more accurately on the fracture-related features.

[0054] Further, the binarization processing is performed on the cropped image to obtain a binary image, and the moments of each order are calculated and the principal axis angle is determined according to the binary image, including taking the average of the pixel points in the marked area as a threshold to perform the binarization processing on the cropped image to obtain the binary image. In view of the problem that the ribs may present different angles of inclination in different slice images, the moments of each order are calculated based on the binary image, and the principal axis angle is calculated to determine the main extension direction of the ribs.

[0055] Specifically, the specific calculation process of calculating the moments of each order based on the binary image and calculating the principal axis angle is as follows:

[0056] First, the zero-order moment is calculated based on the binary image , which is used to determine the total volume (total number of non-zero pixels) of the target area, and the specific formula is as follows:

[0057]

[0058] wherein, is the zero-order moment; is the pixel value of the i-th row and j-th column in the binary image; and are the indexes of the rows and columns of the binary image, respectively; and are the height and width of the binary image, respectively. The vertical first-order moment and the horizontal first-order moment

[0059] are calculated based on the binary image and the rows and columns of the binary image, respectively, which are used to determine the centroid position of the target area, and the specific formula is as follows:

[0060]

[0061]

[0062] wherein, is the vertical first-order moment; is the horizontal first-order moment.

[0063] The vertical second-order moment , the horizontal second-order moment , and the horizontal and vertical second-order moment are calculated, which are used to describe the distribution characteristics of the target area relative to the centroid, and the specific formula is as follows:

[0064]

[0065]

[0066] ​​

[0067] in, It is the vertical second moment; It is the second horizontal moment; It is the second moment of both horizontal and vertical directions.

[0068] The centroid coordinates are determined based on the calculated moments of each order. , ), where the x-coordinate y-axis .

[0069] Finally, through the aforementioned moments and the centroid coordinates ( , Calculate the principal axis angle The specific formula is as follows:

[0070]

[0071] in, The main axis angle.

[0072] It should be noted that the principal axis angle calculation method based on image moments can accurately determine the main extension direction of the ribs in the image, and has good noise resistance and stability. Compared with the direct edge detection method, it can more reliably handle the rib morphological changes and unclear boundaries caused by fractures. The binarization process, through threshold selection, ensures that stable binarization results can be obtained under different CT scanning conditions, providing a reliable foundation for subsequent moment calculation.

[0073] Furthermore, rotating and scaling the cropped image to a uniform size based on the principal axis angle includes rotating the cropped image. The angle is adjusted so that the ribs are basically vertical in the slice image. The rotated image is then scaled proportionally so that the long side is 100 pixels and the short side is symmetrically padded to 100 pixels with a value of 0, resulting in an input image of uniform size of 100×100 pixels, forming a multi-instance slice image.

[0074] By uniformly adjusting the ribs to a vertical position, the problem of inconsistent orientation of ribs in different locations in CT images is eliminated. This allows the network to perform feature learning under a standardized spatial configuration, improving the consistency and comparability of feature extraction. The uniform size processing not only meets the format requirements of the network input, but more importantly, by maintaining the aspect ratio of the image and using zero-value padding, it avoids the deformation distortion that may be caused by direct stretching, ensuring the authenticity of the rib morphological features. This lays a solid data foundation for subsequent multi-scale feature extraction, enabling CT images from different sources and under different scanning conditions to be processed in a unified coordinate system and scale, thus improving the model's generalization ability and prediction accuracy.

[0075] S2: Input the multi-instance slice image into a multi-scale feature extraction network to extract multi-instance features at four different scales.

[0076] In this embodiment, as Figure 3 The diagram shows the overall network structure of this invention, including the stem layer, four feature extraction stages, feature alignment module, packet fusion module, and the connection relationships of the packet regressor. Figure 3 The data flow and information transmission paths between modules are clearly marked, intuitively demonstrating the complete network architecture from multi-instance slice image input to fracture time prediction output, reflecting the design ideas of multi-scale feature extraction and multi-instance fusion. In particular, by integrating multiple preprocessed multi-instance slice images of the same fracture site into a complete data packet input network, the rich feature information of the lesion in different image sections can be fully utilized. First, the original features of each slice image are fully extracted and preliminarily analyzed, and then the features of all slice images are fused to achieve comprehensive judgment.

[0077] Specifically, the multi-instance slice image is input into a multi-scale feature extraction network to extract multi-instance features at four different scales. This includes preliminary feature extraction and size compression of the multi-instance slice image through a stem layer. The image output from the stem layer is then sequentially input into the first, second, third, and fourth feature extraction stages. Each feature extraction stage processes the image output from the stem layer through a pooling attention module, outputting multi-instance features at the corresponding scale, resulting in four multi-instance features at different scales.

[0078] Preliminary feature extraction and size compression of the multi-instance slice images through the stem layer includes, assuming the input image... ,in The number of cases refers to the number of cases for different fracture lesions. The value is variable and is determined based on the number of slices showing the lesion location in the current CT image. Input image After including The stem layer, which incorporates convolution and downsampling, reduces the feature map size and alleviates computational burden. As the initial processing module of the network, the stem layer performs appropriate size compression and preliminary feature abstraction on the input image in the early stages of feature extraction, laying the foundation for subsequent multi-scale processing. Furthermore, the choice of convolution kernel size fully considers the spatial characteristics of the rib structure. The convolutional kernel can effectively capture local features of rib edges and textures. In addition, the feature map dimensionality is compressed through downsampling, which improves the computational efficiency of subsequent processing while maintaining key information, laying the foundation for clinical application scenarios with high real-time requirements.

[0079] The image output by the stem layer is sequentially input into the first feature extraction stage, the second feature extraction stage, the third feature extraction stage, and the fourth feature extraction stage, that is, the image output by the stem layer is sequentially subjected to the feature extraction operations of the four stages to obtain four different scale features. Specifically, the image output by the stem layer is first processed by the pooling attention module to obtain the first scale multi-instance feature Thereafter, the image is sequentially subjected to downsampling and the pooling attention module to obtain the second scale multi-instance feature , the third scale multi-instance feature , and the fourth scale temporary feature . The fourth scale temporary feature is subjected to average pooling and flattening to obtain the fourth scale multi-instance feature .

[0080] It should be noted that, considering that the diversity of fracture damage degree can cause a large difference in feature scale, the network needs to generate features of different scales during feature extraction, and each scale contains multiple instance features. The present embodiment can mine multi-level feature information related to fractures layer by layer from fine granularity to coarse granularity through four ordered feature extraction operations, and each stage focuses on feature extraction of a specific scale, avoiding important information that may be missed by single-scale features.

[0081] Further, in order to fully utilize the characteristics that the fracture site usually has high-density features, focus on the region of interest, and suppress irrelevant noise, the present embodiment introduces a pooling attention module with channel attention and spatial attention mechanisms in the multi-scale feature extraction network to guide the network to focus on the high-density region of the fracture, effectively improving the feature representation capability. As shown in Figure 4 , which is a structural diagram of the pooling attention module, including channel attention and spatial attention, the diagram shows the complete process of input features passing through a convolution layer, a linear layer, a global pooling operation, an excitation operation, and attention weight calculation, as well as the synergistic mechanism of channel attention and spatial attention, highlighting the technical features of the pooling attention module in focusing on the high-density region of the fracture.

[0082] The channel attention includes sequentially passing the input feature F0 through a convolution layer, a linear layer, a Gaussian Error Linear Units (GELU) activation, and a linear layer to obtain an intermediate feature F1; performing global average pooling and global maximum pooling operations on the intermediate feature F1 in the spatial dimension to obtain channel-wise average feature and maximum feature . The average feature and the maximum feature The shared weight excitation operation is performed to fully excavate the dependency between the channels and obtain average features and maximum features of the excitation operation result , the excitation operation comprising: compressing the channel dimension by means of a linear dimension reduction layer with a reduction ratio = 16, passing through a nonlinear activation function ReLU, and then recovering to the original channel dimension of the input feature through a linear dimension increase layer. The excitation operation results of the average features and the maximum features are added element by element, and are calculated through a Sigmoid activation function to obtain channel attention weights . The channel attention weights and the intermediate features F1 are multiplied channel by channel to obtain channel attention output features F2.

[0083] Specifically, the specific formula of the channel attention weights is as follows:

[0084]

[0085] wherein, is the channel attention weight; and are the weight vectors of the dimension reduction layer and the dimension increase layer, respectively.

[0086] Considering that the fracture injury site is often accompanied by callus formation, which will increase the bone density of the corresponding area, and the callus morphology is the key basis for judging the fracture time, therefore, in order to effectively focus on the callus area detail features, so that the model can more accurately capture the bone density changes and callus morphology, and strengthen the fracture time discrimination ability, the spatial attention is used to further process the channel attention output features F2.

[0087] The spatial attention comprises: adopting average pooling processing on the multi-instance slice image in the spatial dimension, adjusting the multi-instance slice image to the same spatial scale as the channel attention output features F2, and then normalizing by using a softmax function to generate spatial attention weights . The spatial attention weights and the channel attention output features F2 are multiplied pixel by pixel to obtain the final output features of the pooling attention module.

[0088] To solve the problem that the rib fracture area is small and the boundary is fuzzy relative to the whole CT image volume, and the traditional method is difficult to accurately locate and extract the fracture-related features, a multi-scale feature extraction network combined with a pooling attention mechanism is proposed. By introducing a combination mechanism of channel attention and spatial attention, the formation area of bone callus and the bone density change characteristics can be identified and focused. Through four ordered stages of feature extraction operations, from fine-grained bone callus microstructure to coarse-grained bone fracture overall profile, a complete feature hierarchy from micro to macro is established. The pooling attention module uses the synergistic effect of global average pooling and maximum pooling to retain the global statistical information of the features and highlight the detailed features of the local key areas. Based on the physiological characteristics that bone callus formation in the fracture injury site will increase the bone density of the area, the spatial attention mechanism can accurately locate the corresponding high-density area and effectively suppress the interference of irrelevant background information. This makes the network accurately capture multi-level features related to fracture time while maintaining computational efficiency, providing high-quality feature representation for subsequent time prediction.

[0089] S3: performing feature alignment processing on the multi-instance features of the first three scales to obtain aligned multi-instance features, and generating instance evidence values based on the multi-instance features of the fourth scale, and calculating the contribution weight of the instance according to the instance evidence values.

[0090] To solve the problem of spatial consistency and contribution difference of different instance features in multi-instance learning, on the one hand, the multi-instance features of the first three scales are aligned to solve the possible feature shift problem between instances; on the other hand, instance evidence values are generated based on the multi-instance features of the fourth scale to provide reliable weight basis for subsequent feature fusion.

[0091] Specifically, the feature alignment processing on the multi-instance features of the first three scales to obtain the aligned multi-instance features includes inputting the multi-instance features of the first three scales into a feature alignment module respectively. The multi-instance features of the first three scales are processed by feature offset learning and alignment based on the offset through deformable convolution. The features after deformable convolution are sequentially processed by layer normalization and linear transformation to obtain the aligned multi-instance features.

[0092] In the same fracture lesion of multiple slice images, the fracture site will shift slightly due to the difference between slice images and the uncertainty in the manual framing process, so it is necessary to align the features of different slice images. By inputting the multi-instance features of the first three scales into the feature alignment module for inter-instance alignment, the aligned multi-instance features are obtained. 、 and are input into the feature alignment module for inter-instance alignment, and the aligned multi-instance features are obtained, wherein Figure 5The feature alignment module is shown in the structural diagram, and is composed of a deformable convolution, a layer normalization, a linear layer, a ReLU activation function and a linear layer, for realizing accurate alignment between features, and obtaining multi-instance features after alignment.

[0093] The feature offset learning and offset-based alignment of the multi-instance features of the first three scales by the deformable convolution are core links of feature alignment; the deformable convolution can adjust the sampling position of the convolution kernel according to the spatial distribution characteristics of the input features, so as to learn the spatial offset mode between different instances; compared with the traditional fixed convolution kernel, the deformable convolution can better adapt to the position change of the fracture part in different slices, realize implicit alignment between features, and effectively eliminate the spatial inconsistency problem caused by the image acquisition angle and manual frame selection error factors through accurate modeling of the feature offset.

[0094] Secondly, in the layer normalization and linear transformation processing of the features after the deformable convolution, the layer normalization operation ensures the consistency of different instance features in the numerical distribution, avoiding the influence of amplitude difference on the subsequent processing effect. Linear transformation further adjusts the representation space of the features, so that the aligned features are more suitable for subsequent fusion operation, ensuring the accuracy and stability of feature alignment, and laying a foundation for high-quality feature fusion.

[0095] Further, the multi-instance feature of the fourth scale generates an instance evidence value, and the contribution weight of the instance is calculated according to the instance evidence value, which includes inputting the multi-instance feature of the fourth scale into a multi-layer perception machine to output a distribution parameter. According to the evidence theory, the evidence value corresponding to each instance is calculated according to the distribution parameter. The distribution parameter is explicitly constrained by an evidence loss function. The evidence value corresponding to each instance is normalized to obtain the contribution weight of each instance.

[0096] Inputting the multi-instance feature of the fourth scale into a multi-layer perception machine to output a distribution parameter means that for a regression task facing a continuous label, the fourth scale multi-instance feature is input into a multi-layer perception machine to output a distribution parameter. Inputting the multi-layer perception machine for processing, outputting a distribution parameter, providing a theoretical basis for subsequent evidence value calculation. Specifically, assuming that the prediction target variable y is independent and identically distributed and subject to a Gaussian distribution , the mean obeys a Gaussian distribution , the variance obeys an inverse gamma distribution , where is the mean parameter of the mean distribution, is the scale parameter of the mean distribution, is the shape parameter of the variance distribution, is a scale parameter of the variance distribution. The Gaussian distribution is predicted by the multi-layer perception and inverse gamma distribution Four groups of parameters in the inverse gamma distribution, that is 、 、 and .

[0097] Based on the evidence theory, the evidence value corresponding to each instance is calculated according to the distribution parameter , and the specific formula is as follows:

[0098]

[0099] wherein, is the evidence value; is a scale parameter of the mean distribution; is a shape parameter of the variance distribution.

[0100] It should be noted that the size of the evidence value directly reflects the credibility of the corresponding instance, and then reflects the contribution size of each slice image in the model decision process; the larger the evidence value, the stronger the support of the corresponding instance to the final prediction result, and a higher weight should be given in the feature fusion process.

[0101] The explicit constraint of the distribution parameter by the evidence loss function means that the evidence value is explicitly constrained by the corresponding loss function, and then the feasible contribution of each instance to the regression target is reflected, avoiding the randomness and instability of weight allocation, which is a key step to ensure the reliability of the evidence value. The distribution parameter is optimized by minimizing the evidence loss function, and the specific formula of the evidence loss function is as follows:

[0102]

[0103] wherein, is the evidence loss function; is the Gamma function.

[0104] The evidence value corresponding to each instance is normalized to obtain the contribution weight of each instance, that is After the evidence value is processed by the softmax normalization, the contribution weight of each instance is obtained , ensuring that the sum of all weights is 1, which meets the basic requirements of probability distribution. By performing normalization processing, not only the rationality of the weight is ensured, but also the importance comparison between different instances is more intuitive and interpretable, which provides a reliable quantitative basis for subsequent feature fusion.

[0105] To solve the problem of feature space inconsistency caused by scanning accuracy difference and artificial labeling uncertainty in multiple slice images of the same fracture lesion, a feature alignment module based on deformable convolution and an instance weight calculation method based on evidence theory are proposed. The feature offset learning through deformable convolution can adjust the sampling position of the convolution kernel and learn the spatial offset mode between different instances, which has stronger adaptability than traditional fixed convolution kernel. It can effectively eliminate the spatial inconsistency caused by factors such as image acquisition angle and artificial labeling error, and realize accurate alignment at the feature level. At the same time, the weight calculation based on evidence theory introduces statistical principles into the multi-instance learning framework by predicting four groups of parameters of Gaussian distribution and inverse gamma distribution, and calculates the evidence value of each instance to quantify the reliability. Compared with traditional average pooling or simple attention mechanism, the present application can more accurately reflect the actual contribution of each slice image in model decision making, and through the constraint of evidence loss function, it ensures the stability and interpretability of weight distribution, providing a reliable quantitative basis for subsequent feature fusion.

[0106] S4: According to the contribution weight, the aligned multi-instance features and the multi-instance features of the fourth scale are respectively fused to obtain four package features of different scales.

[0107] In the present embodiment, the obtained contribution weight is used to differentially process and fuse different instance features. The aligned multi-instance features of the first three scales 、 and are respectively aligned between instances through the feature alignment module, and then input into the package fusion module and the pooling attention module to obtain the package features of the first three scales. The operation of the pooling attention module is to convert the spatial dimension of the feature to 1 after convolution of the input feature and the convolution kernel with the same size as the input feature. In addition, for the multi-instance features of the fourth scale , on the one hand, the fourth scale package feature is obtained directly through the package fusion module, and on the other hand, the evidence value is calculated and generated after the multi-layer perceptron. Then, the contribution weight of each instance is obtained by softmax calculation. The contribution weight of each instance is input into the package fusion module of each scale for fusion of instance features.

[0108] Specifically, according to the contribution weight, the aligned multi-instance feature and the multi-instance feature of the fourth scale are respectively subjected to feature fusion to obtain four package features of different scales, including sorting the contribution weights according to the size, determining the instance with the highest evidence value as the main instance, and the remaining instances as reference instances. The main instance feature and the reference instance feature are subjected to deep interaction integration through an instance feature interaction module. The main instance feature and the interaction-integrated reference instance feature are weighted and accumulated according to the contribution weights. The steps from sorting the contribution weights according to the size to the weighted accumulation are performed on the aligned multi-instance feature and the multi-instance feature of the fourth scale respectively to obtain four package features of different scales.

[0109] The sorting of the contribution weights according to the size includes, for the evidence value The contribution weights of each instance obtained after softmax normalization The contribution weights of each instance are sorted from large to small The contribution weights of each instance are sorted from large to small The instance with the highest evidence value, i.e. The corresponding instance is defined as the main instance, which plays a leading role in the fracture prediction result; the remaining The corresponding instances are defined as reference instances, which are used to supplement the main instance information.

[0110] Further, as Figure 6 The structure diagram of the package fusion module is shown, including the division of the main instance and the reference instance, the instance feature interaction process and the weighted fusion operation, which is used to illustrate how the main instance and the reference instance are subjected to deep interaction through the attention mechanism, and finally realize intelligent feature fusion based on credibility.

[0111] Specifically, when the main instance feature and the reference instance feature are subjected to deep interaction integration through the instance feature interaction module, the main instance feature is represented as , and the reference instance feature is represented as (k is the reference instance number). Since the main instance feature itself has the most core leading feature, it is not necessary to have additional interaction operation. However, for the reference instance feature, in order to effectively fuse the supplementary information contained in the reference instance feature with the main instance, the reference instance feature needs to be input into the instance feature interaction module together with the main instance feature, so as to mine the feature correlation between instances.

[0112] Secondly, as Figure 7The example feature interaction module is shown, including the introduction of position encoding, the generation process of query vector Q, key vector K, and value vector V, and the specific operation of attention weight calculation and feature fusion. In the interactive attention operation link, the input is the main instance feature and a reference instance feature, and the learnable position encoding is introduced, which is added to the main instance and the reference instance feature respectively, so that the model can pay attention to the content feature and the spatial layout feature at the same time when interacting. The main instance feature generates the query vector Q through the linear layer, and the reference instance feature generates the key vector K and the value vector V through two independent linear layers respectively. The similarity between the query vector Q and the key vector K is calculated through attention operation, and then the value vector V is weighted and fused according to the similarity to obtain the fused feature, realizing the deep interactive integration of the main instance feature and the reference instance feature.

[0113] The specific formula of the deep interactive integration of the main instance feature and the reference instance feature is as follows:

[0114] ;

[0115] Wherein, FF is the instance feature interaction operation; d is the dimension of the key vector.

[0116] It should be noted that, in order to ensure the stability and consistency of the feature interaction process, the instance feature interaction module uniformly adopts shared weights for all instances, so that the reference instance can adjust the corresponding feature representation based on the relevance degree with the main instance, thereby realizing more accurate and targeted feature supplement.

[0117] Further, in order to better retain the discriminative information of the key instance and reasonably utilize the supplement value of the reference instance, the quality and prediction performance of the fused feature are improved, the main instance feature and the interactive reference instance feature are weighted and accumulated according to the instance contribution weight, and finally the bag feature is obtained, and the specific formula is as follows:

[0118] ;

[0119] Wherein, is the bag feature; FF is the instance feature interaction operation.

[0120] For the aligned multi-instance features of the first three scales, the above-mentioned contribution weight ordering, main parameter instance determination, feature interaction, and weighted accumulation processes are performed respectively to obtain the corresponding scale package features. For the fourth scale multi-instance feature, the corresponding processing measures are the same as those of the aligned multi-instance features of the first three scales, except that the spatial dimension of the fourth scale multi-instance feature has been compressed to 1, so that no pooling processing is required after fusion. Through this unified processing framework, the multi-instance features of the four different scales can be fused based on the evidence value while maintaining their respective scale characteristics, ultimately obtaining package features of four different scales, which not only retains the advantages of multi-scale feature extraction, but also ensures the accuracy and discriminability of feature representation through the evidence theory-guided fusion strategy, providing high-quality feature input for subsequent fracture time prediction.

[0121] To solve the problem that the existing multi-instance learning method simply averages the fusion when facing a continuous label regression task, the instances are divided into main instances and reference instances according to the evidence value, ensuring that the most discriminative slice images play a leading role in the fusion process and avoiding the dilution of key information caused by simple average fusion. Through the attention mechanism, the main instances and reference instances are deeply integrated, allowing the reference instances to adjust the corresponding feature representation based on the relevance to the main instances, thereby providing more accurate and targeted supplementary information. In the weighted accumulation process, the contribution of each instance is proportional to its evidence value, embodying the feature fusion idea based on credibility. To ensure the stability and consistency of the feature interaction process, the instance feature interaction module uniformly uses a shared weight mechanism for all instances, which not only ensures the dominance of key instances but also fully utilizes the supplementary value of other instances. Compared with the traditional equal-weight average or maximum pooling method, it can better retain the discriminative information of key instances and improve the quality of fused features and prediction performance.

[0122] S5: inputting the four different scale package features after splicing into a package regressor to obtain a prediction result of the rib fracture time.

[0123] Specifically, inputting the four different scale package features after splicing into a package regressor to obtain a prediction result of the rib fracture time includes performing a splicing operation on the four different scale package features in the feature dimension to form the final feature for fracture time regression, wherein splicing the four different scale package features in the feature dimension can fully integrate multi-level feature information from fine granularity to coarse granularity, providing comprehensive feature input for subsequent regression prediction.

[0124] The final fusion feature is input into a package regressor for processing, where the package regressor is composed of a multilayer perceptron and a Sigmoid activation function. The multilayer perceptron is used to learn the mapping relationship between the features and the fracture time through nonlinear transformation, and the Sigmoid activation function ensures that the fracture time prediction result is in the interval of 0 to 1.

[0125] The fracture time prediction result is multiplied by the actual fracture day maximum value to obtain the predicted fracture day. In an embodiment, the actual fracture day maximum value is 90, and the normalized prediction value is converted into a day representation with actual clinical significance through linear transformation to obtain the final prediction result of the rib fracture time.

[0126] It should be noted that the mean absolute error loss is used as the loss function in the training process of the above rib fracture time prediction. The mean absolute error loss is a commonly used loss function in regression tasks, which measures the model performance by calculating the average absolute error between the predicted value and the true value. The calculation process is simple and efficient, and has strong robustness to abnormal values, and will not cause the error to be excessively enlarged due to extreme values; the specific formula of the mean absolute error loss is as follows:

[0127]

[0128] Wherein, N is the number of training samples, is the true value of the t-th sample, is the prediction value of the t-th sample.

[0129] The total loss function is the sum of the mean absolute error loss and the evidence loss

[0130]

[0131] Wherein, is the total loss function.

[0132] ​To solve the problem that single scale feature cannot cover all the judgment basis for fracture time judgment which needs to integrate multi-level information, four different scale bag features are spliced in feature dimension to form a complete feature representation containing from macro fracture contour to micro callus details, avoiding information loss caused by simple feature addition or average fusion. The bag regressor is designed by combining multilayer perceptron and Sigmoid activation function, which learns the mapping relationship between fused features and fracture time through deep nonlinear transformation, has stronger expression ability than traditional linear regression method, and can capture the nonlinear change rule in fracture healing process. The Sigmoid activation function ensures that the output value is limited in the range of zero to one, which is convenient for subsequent conversion to actual days. In the model training process, the joint optimization strategy of mean absolute error loss and evidence loss is adopted to ensure the optimization of prediction accuracy and the rationality of instance weight distribution.

[0133] In another aspect, the embodiment also provides a rib fracture time prediction system based on multi-instance learning, which comprises:

[0134] An image preprocessing module acquires CT image data of rib fracture, pre-processes the CT image data, and obtains multi-instance slice images.

[0135] A feature extraction module inputs the multi-instance slice images into a multi-scale feature extraction network to extract four different scale multi-instance features respectively.

[0136] A feature alignment module performs feature alignment processing on the multi-instance features of the first three scales to obtain aligned multi-instance features, generates instance evidence values based on the multi-instance features of the fourth scale, and calculates the contribution weight of the instance according to the instance evidence values.

[0137] A bag fusion module performs feature fusion on the aligned multi-instance features and the multi-instance features of the fourth scale respectively according to the contribution weight to obtain four different scale bag features.

[0138] A time prediction module inputs the four different scale bag features after splicing into a bag regressor to obtain the prediction result of rib fracture time.

[0139] If the above functions are realized in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or parts of the present application that make contributions to the prior art or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0140] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered a list of executable instructions for implementing logic functions, and can be specifically embodied in any computer-readable medium for use by an instruction execution system, apparatus, or device, such as a computer-based system, a processor-based system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions, or in conjunction with such an instruction execution system, apparatus, or device. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device.

[0141] More specific examples (a non-exhaustive list) of the computer-readable medium include the following: an electrical connection having one or more wires (electrical devices), a portable computer diskette (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CD ROM). In addition, the computer-readable medium can even be paper or other suitable medium on which the program can be printed, as the program can be electronically obtained, for example, by optical scanning of the paper or other medium, followed by electronic editing, interpretation, or necessary processing, and then stored in a computer memory if necessary. Other suitable media can also be used.

[0142] It should be understood that portions of the application can be implemented in hardware, software, firmware, or combinations thereof. In the embodiments described above, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, and as in another embodiment, any of the following technologies known in the art or a combination thereof can be used: discrete logic circuitry having logic gates for implementing logic functions on data signals, application specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), etc.

[0143] Example 2, which is an embodiment of the present application, provides a rib fracture time prediction method based on multi-instance learning. In order to verify the beneficial effects of the present application, economic benefit calculation and simulation experiments are used for scientific demonstration.

[0144] In order to verify the effectiveness of the rib fracture time prediction method based on multi-instance learning proposed in the present application, the private data set used in the experiment contains 350 three-dimensional chest CT image data from multiple centers, and is equipped with corresponding injury time information; The rib region is labeled by forensic experts, and a total of 1864 rib fracture lesions are involved; The samples are divided into training set, validation set and test set in the ratio of 8:1:1, and at the same time, it is strictly ensured that all image data of the same patient only appears in the same data subset, effectively avoiding the risk of data leakage.

[0145] First, the mean absolute error MAE and the determination coefficient R 2 As an evaluation index of the regression task, the prediction performance of the model is comprehensively measured, and the specific formula is as follows:

[0146]

[0147]

[0148]

[0149]

[0150] Wherein, M is the number of test samples; is the true value of the t th sample; is the predicted value of the t th sample; is the mean of the true label; SST is the total sum of squares; SSE is the residual sum of squares.

[0151] Secondly, in order to verify the effectiveness of each key module of the application, the application carries out corresponding ablation experiments on the pooling attention, multi-scale feature and multi-instance evidence fusion method; in addition, in the comparative experiment, the method of the application is compared with other networks based on multi-instance learning, including Mean-pooling, Max-pooling, ABMIL, TransMIL and DSMIL, verifying the effectiveness of the method proposed in the application; wherein the results of the ablation experiment are shown in Table 1, and the results of the comparative experiment are shown in Table 2.

[0152] Table 1 Ablation experiment results of multi-instance rib fracture time prediction

[0153]

[0154] Table 2 Comparative experiment results of multi-instance rib fracture time prediction

[0155]

[0156] As can be seen from Table 1, compared with the baseline network (ABMIL), the method of the application has improved in two key indicators, that is, the mean absolute error MAE is reduced from 9.59 days to 8.50 days, reduced by 1.09 days; the determination coefficient R2 is improved from 0.640 to 0.714, improved by 0.074, and the optimization of these indicators fully embodies the performance advantage of the model in the fracture time prediction task; in addition, as can be seen from Table 2, the method of the application has improved in each evaluation indicator compared with the existing multi-instance learning method, fully verifying that the multi-instance learning network proposed in the application can better realize the accurate prediction of the fracture time.

[0157] It should be noted that the above embodiments are only used to illustrate the technical solutions of the application and not to limit it, although the application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the application can be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the application, and they should be covered in the scope of the claims of the application.

Claims

1. A method for predicting the time of rib fracture based on multiple instance learning, characterized in that, The method comprises the following steps: obtaining CT image data of a rib fracture, preprocessing the CT image data to obtain a multi-instance slice image; inputting the multi-instance slice image into a multi-scale feature extraction network to extract four multi-instance features of different scales respectively; performing feature alignment processing on the multi-instance features of the first three scales to obtain aligned multi-instance features, generating instance evidence values based on the multi-instance features of the fourth scale, and calculating the contribution weights of the instances according to the instance evidence values; performing feature fusion on the aligned multi-instance features and the multi-instance features of the fourth scale respectively according to the contribution weights to obtain four package features of different scales; inputting the four package features of different scales into a package regressor after splicing to obtain a prediction result of the rib fracture time; the step of inputting the multi-instance slice image into a multi-scale feature extraction network to extract four multi-instance features of different scales respectively comprises the following steps: performing preliminary feature extraction and size compression on the multi-instance slice image through a stem layer, wherein the stem layer comprises a 4*4 convolution layer and a down-sampling process; inputting the image output by the stem layer into a first feature extraction stage, a second feature extraction stage, a third feature extraction stage and a fourth feature extraction stage in sequence, each feature extraction stage processing the image output by the stem layer through a pooling attention module to output multi-instance features of a corresponding scale, and obtaining four multi-instance features of different scales; the step of performing feature alignment processing on the multi-instance features of the first three scales to obtain aligned multi-instance features comprises the following steps: inputting the multi-instance features of the first three scales into a feature alignment module respectively; performing feature offset learning on the multi-instance features of the first three scales through deformable convolution; performing layer normalization and linear transformation processing on the features after feature offset learning in sequence to obtain aligned multi-instance features; the step of generating instance evidence values based on the multi-instance features of the fourth scale and calculating the contribution weights of the instances according to the instance evidence values comprises the following steps: inputting the multi-instance features of the fourth scale into a multi-layer perception to output four groups of distribution parameters; calculating the evidence value corresponding to each instance based on evidence theory according to the four groups of distribution parameters; performing explicit constraint on the four groups of distribution parameters through an evidence loss function; performing normalization processing on the evidence value corresponding to each instance to obtain the contribution weight of each instance; the step of performing feature fusion on the aligned multi-instance features and the multi-instance features of the fourth scale respectively according to the contribution weights to obtain four package features of different scales comprises the following steps: sorting the contribution weights according to their sizes, determining the instance with the highest evidence value as the main instance, and determining the remaining instances as reference instances; performing deep interaction integration on the main instance feature and the reference instance feature through an instance feature interaction module; weighting and accumulating the main instance feature and the reference instance feature after interaction according to the contribution weights; performing the steps from sorting the contribution weights according to their sizes to weighting and accumulating on the aligned multi-instance features and the multi-instance features of the fourth scale respectively to obtain four package features of different scales.

2. The multi-instance learning based rib fracture time prediction method of claim 1, wherein: The CT image data is preprocessed to obtain a multi-instance slice image, including marking a rib region in the CT image data, determining a minimum circumscribed rectangle containing the rib region and performing cropping; A binary image is obtained by performing binaryzation processing on the cropped image, and each order moment is calculated according to the binary image and a principal axis angle is determined; The cropped image is rotated and corrected according to the principal axis angle and scaled to a uniform size to obtain a multi-instance slice image.

3. The multi-instance learning based rib fracture time prediction method of claim 2, wherein: The four different scale bag features are spliced to input a bag regressor to obtain a prediction result of rib fracture time, including performing a splicing operation on the four different scale bag features in the feature dimension to form a final fusion feature; The final fusion feature is input into the bag regressor for processing, wherein the bag regressor is composed of a multilayer perceptron and an activation function, and outputs a fracture time prediction value in a preset interval range; The fracture time prediction value is converted into actual fracture days to obtain a prediction result of rib fracture time.

4. A rib fracture time prediction system based on multi-instance learning using the method of any one of claims 1-3, characterized in that: An image preprocessing module acquires CT image data of rib fracture, pre-processes the CT image data, and obtains a multi-instance slice image; A feature extraction module inputs the multi-instance slice image into a multi-scale feature extraction network to extract four different scale multi-instance features; A feature alignment module performs feature alignment processing on the first three scale multi-instance features to obtain aligned multi-instance features, generates instance evidence values based on the fourth scale multi-instance features, and calculates the contribution weight of the instance according to the instance evidence values; A bag fusion module performs feature fusion on the aligned multi-instance features and the fourth scale multi-instance features according to the contribution weight to obtain four different scale bag features; A time prediction module splices the four different scale bag features to input a bag regressor to obtain a prediction result of rib fracture time.

5. A computer device comprising: A memory and a processor; The memory stores a computer program, and the processor executes the computer program to implement the steps of the method of any one of claims 1-3.

6. A computer readable storage medium having stored thereon a computer program, characterized in that: The computer program is executed by the processor to implement the steps of the method of any one of claims 1-3.

Images