Rib fracture time prediction method and system based on multi-instance learning

Abstract

The invention relates to the technical field of image recognition, and discloses a rib fracture time prediction method and system based on multi-instance learning, and the method comprises the steps: obtaining the CT image data of rib fracture, and carrying out the preprocessing of the CT image data, and obtaining a multi-instance slice image; inputting the multi-instance slice image into a multi-scale feature extraction network, and respectively extracting multi-instance features of four different scales; performing feature alignment processing on the multi-instance features of the first three scales to obtain aligned multi-instance features, generating instance evidence values based on the multi-instance features of the fourth scale, and calculating contribution weights of instances according to the instance evidence values; according to the contribution weight, performing feature fusion on the aligned multi-instance features and the multi-instance features of the fourth scale to obtain four package features of different scales; and splicing the four package features with different scales, and inputting the spliced package features into a package regression device to obtain a prediction result of the rib fracture time. Accurate prediction of fracture time is realized, and the method does not depend on a fixed number of images.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image recognition, in particular to a rib fracture time prediction method and system based on multi-instance learning. BACKGROUND

[0002] With the development of forensic identification technology and the refinement of judicial practice needs, the accurate inference of rib fracture injury time has become an important technical challenge in the field of forensic medicine. Rib fracture, as a common injury type in forensic identification, is directly related to the accurate judgment of the formation time, which is related to the causality determination between fracture and trauma and the qualitative analysis of the case. In particular, in the case of old fracture at the fracture site and relatively concealed fracture, accurate time prediction is particularly crucial. Chest computed tomography (CT) can clearly show key anatomical features such as discontinuity of bone cortex, fracture line direction and fracture end displacement due to its high spatial resolution and three-dimensional reconstruction technology advantages, and has become the gold standard for trauma assessment.

[0003] However, the existing technology still has deficiencies in rib fracture time prediction. First, in the identification of human body injury degree, due to the urgent time limit for handling cases, the rib fracture area is smaller in volume and the boundary is blurred compared to the whole CT image. Artificial screening is time-consuming and laborious, and subtle fractures are easy to be missed. Current research on rib fracture injury time based on imageomics technology is less, and existing methods mostly rely on CT images for local labeling and feature extraction, but most of them are limited to rough binary classification discrimination, which is difficult to meet the demand of accurate time prediction.

[0004] Secondly, existing research uses multi-scale and cyclic convolution networks to classify and predict the fracture category and time, but the time interval is usually set to more than 15 days, with a large span and insufficient accuracy. More importantly, due to the need for a fixed number of CT two-dimensional slice images as input for each fracture, data resampling is required, and due to the difference in fracture injury degree and acquisition equipment, noise is inevitably introduced during the resampling process, affecting the accuracy of time prediction.

[0005] In addition, although the multi-instance feature fusion method can reduce the dependence on the fixed number of slice images, existing methods such as average pooling, maximum pooling and gate attention mechanism-based methods are mainly for classification tasks, and the core assumption is that only a few key instances within the bag play a dominant role in the label. However, in the regression task facing continuous labels, all instances tend to contribute to the prediction result to varying degrees, and the contribution degree is greatly affected by feature saliency and image clarity. Therefore, there is a significant difference between the design assumption of the multi-instance method in the classification task and the actual situation in time prediction. SUMMARY

[0006] In view of the above problems, the present application is proposed.

[0007] To solve the above technical problems, the present application provides the following technical solutions: a rib fracture time prediction method based on multi-instance learning, comprising: Obtaining CT image data of rib fracture, pre-processing the CT image data to obtain a multi-instance slice image; Inputting the multi-instance slice image into a multi-scale feature extraction network to extract four multi-instance features of different scales respectively; Performing feature alignment processing on the multi-instance features of the first three scales to obtain aligned multi-instance features, generating instance evidence values based on the multi-instance features of the fourth scale, and calculating the contribution weight of the instance according to the instance evidence values; According to the contribution weight, the aligned multi-instance features and the multi-instance features of the fourth scale are respectively fused to obtain four package features of different scales; After splicing the four package features of different scales, inputting them into a package regressor to obtain the prediction result of the rib fracture time.

[0008] As a preferred scheme of the rib fracture time prediction method based on multi-instance learning, wherein: the pre-processing of the CT image data to obtain a multi-instance slice image includes labeling the rib region in the CT image data, determining the minimum circumscribed rectangle containing the rib region and performing cropping; Performing binaryzation processing on the cropped image to obtain a binary image, calculating the moments of each order according to the binary image and determining the principal axis angle; According to the principal axis angle, the cropped image is rotated and corrected and scaled to a uniform size to obtain a multi-instance slice image.

[0009] As a preferred scheme of the rib fracture time prediction method based on multi-instance learning, wherein: inputting the multi-instance slice image into a multi-scale feature extraction network to extract four multi-instance features of different scales respectively includes performing preliminary feature extraction and size compression on the multi-instance slice image through a stem layer, wherein the stem layer includes Convolution layer and down-sampling processing; The image output by the stem layer is sequentially input into the first feature extraction stage, the second feature extraction stage, the third feature extraction stage and the fourth feature extraction stage, each feature extraction stage processes the image output by the stem layer through a pooling attention module to output multi-instance features of corresponding scales, and four multi-instance features of different scales are obtained, wherein the pooling attention module includes channel attention and spatial attention.

[0010] As a preferred scheme of the rib fracture time prediction method based on multi-instance learning provided in the application, wherein: the feature alignment processing of the multi-instance features of the first three scales to obtain the aligned multi-instance features comprises inputting the multi-instance features of the first three scales into a feature alignment module respectively; The feature offset learning and offset-based alignment of the multi-instance features of the first three scales are performed through deformable convolution. The aligned multi-instance features are sequentially subjected to layer normalization and linear transformation processing to obtain the aligned multi-instance features.

[0011] As a preferred scheme of the rib fracture time prediction method based on multi-instance learning provided in the application, wherein: the instance evidence value is generated based on the multi-instance features of the fourth scale, and the contribution weight of the instance is calculated according to the instance evidence value comprises inputting the multi-instance features of the fourth scale into a multi-layer perception machine to output distribution parameters; The evidence value corresponding to each instance is calculated according to the distribution parameters based on evidence theory; The distribution parameters are subjected to explicit constraint through an evidence loss function; The contribution weight of each instance is obtained by normalizing the evidence value corresponding to each instance.

[0012] As a preferred scheme of the rib fracture time prediction method based on multi-instance learning provided in the application, wherein: the aligned multi-instance features and the multi-instance features of the fourth scale are respectively subjected to feature fusion according to the contribution weight to obtain four different scale bag features, which comprises sorting the contribution weight according to the size, determining the instance with the highest evidence value as the main instance, and the remaining instances as reference instances; The main instance feature and the reference instance feature are deeply interacted and integrated through an instance feature interaction module; The main instance feature and the reference instance feature after the interaction and integration are weighted and accumulated according to the contribution weight; The steps from sorting the contribution weight according to the size to the weighted accumulation step are performed on the aligned multi-instance features and the multi-instance features of the fourth scale respectively to obtain four different scale bag features.

[0013] As a preferred scheme of the rib fracture time prediction method based on multi-instance learning provided in the application, wherein: the four different scale bag features are spliced and input into a bag regressor to obtain the prediction result of the rib fracture time, which comprises performing splicing operation on the four different scale bag features in the feature dimension to form the final fusion feature; The final fusion feature is input into a bag regressor for processing, wherein the bag regressor is composed of a multi-layer perception machine and an activation function, and outputs the fracture time prediction value in a preset interval range. The fracture time prediction value is converted into actual fracture days to obtain a prediction result of the rib fracture time.

[0014] A rib fracture time prediction system based on multi-instance learning, comprising: An image preprocessing module, which acquires CT image data of a rib fracture, pre-processes the CT image data, and obtains multi-instance slice images; A feature extraction module, which inputs the multi-instance slice images into a multi-scale feature extraction network to extract multi-instance features of four different scales respectively; A feature alignment module, which performs feature alignment processing on the multi-instance features of the first three scales to obtain aligned multi-instance features, generates instance evidence values based on the multi-instance features of the fourth scale, and calculates the contribution weights of the instances according to the instance evidence values; A bag fusion module, which performs feature fusion on the aligned multi-instance features and the multi-instance features of the fourth scale respectively according to the contribution weights to obtain bag features of four different scales; A time prediction module, which inputs the bag features of the four different scales after splicing into a bag regressor to obtain a prediction result of the rib fracture time.

[0015] A computer readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method of any one of the present application.

[0016] The application provides a rib fracture time prediction method based on multi-instance learning. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0018] Figure 1 The overall flowchart of a rib fracture time prediction method based on multi-instance learning provided by the first embodiment of the present application.

[0019] Figure 2 The data preprocessing flowchart provided by the first embodiment of the present application.

[0020] Figure 3 The structure diagram of the overall network of the first embodiment of the present application.

[0021] Figure 4 The structure diagram of the pooling attention module provided by the first embodiment of the present application.

[0022] Figure 5 The structure diagram of the feature alignment module provided by the first embodiment of the present application.

[0023] Figure 6A structure diagram of a package fusion module provided for a first embodiment of the application.

[0024] Figure 7 A structure diagram of an example feature interaction module provided for the first embodiment of the application. DETAILED DESCRIPTION

[0025] In order to make the above objectives, characteristics and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should belong to the protection scope of the present application.

[0026] Embodiment 1, with reference to Figures 1-7 For an embodiment of the present application, a rib fracture time prediction method based on multi-instance learning is provided, comprising: S1: Obtain CT image data of rib fracture, and pre-process the CT image data to obtain a multi-instance slice image.

[0027] In this embodiment, as Figure 2 The data pre-processing flowchart is shown, and the process of pre-processing the CT image data includes rib region annotation and cutting, binarization processing and image matrix calculation, and rotation correction and size standardization, aiming to eliminate irrelevant information interference in the image, unify the spatial direction and size specification of the image, and provide standardized input data for subsequent multi-instance feature extraction.

[0028] Specifically, the rib region is annotated in the CT image data, and the minimum circumscribed rectangle containing the rib region is determined and cut, including manually annotating the rib region of interest in the original CT image, wherein a quadrilateral containing the entire rib region is used for annotation; considering the irregularity of the manually annotated region, in order to adapt to network input, finding the minimum circumscribed rectangle containing the annotated region and having two sides parallel to the image coordinates, cutting out the corresponding region and setting the pixel value outside the annotated region to 0.

[0029] By using the cutting strategy of the minimum circumscribed rectangle, a large amount of irrelevant background information in the CT image can be effectively removed, the computing resources are concentrated in the key area of rib fracture, and the efficiency and accuracy of subsequent feature extraction are improved. At the same time, by setting the pixel value outside the annotated region to 0, the contrast between the region of interest and the background is further enhanced, which is conducive to the network focusing more accurately on the fracture-related features.

[0030] Further, the binarization processing is performed on the cropped image to obtain a binary image, and the moments of each order are calculated and the principal axis angle is determined according to the binary image, including taking the average of the pixel points in the marked region as a threshold to perform the binarization processing on the cropped image to obtain the binary image. In view of the problem that the ribs may present different angles of inclination in different slice images, the moments of each order are calculated based on the binary image, and the principal axis angle is calculated to determine the main extension direction of the ribs.

[0031] Specifically, the specific calculation process of calculating the moments of each order based on the binary image and calculating the principal axis angle is as follows: First, the zero-order moment is calculated based on the binary image , which is used to determine the total volume (total number of non-zero pixels) of the target region, and the specific formula is as follows:

[0032] wherein, is the zero-order moment; is the pixel value in the i-th row and j-th column of the binary image; are the indexes of the rows and columns of the binary image, respectively; and are the height and width of the binary image, respectively.

[0033] The vertical first-order moment and the horizontal first-order moment are calculated based on the binary image and the rows and columns of the binary image, respectively, which are used to determine the centroid position of the target region, and the specific formula is as follows:

[0034]

[0035] wherein, is the vertical first-order moment; is the horizontal first-order moment.

[0036] The vertical second-order moment , the horizontal second-order moment , and the horizontal and vertical second-order moment are calculated, which are used to describe the distribution characteristics of the target region relative to the centroid, and the specific formula is as follows:

[0037]

[0038]

[0039] wherein, is the vertical second-order moment; ​​​horizontal second moment; horizontal and vertical second moment.

[0040] Based on the calculated moments of each order to determine the centroid coordinates , , wherein the horizontal coordinate , the vertical coordinate .

[0041] Finally, the principal axis angle , is calculated by the moments of each order and the centroid coordinates , and the specific formula is as follows:

[0042] wherein, is the principal axis angle.

[0043] It should be noted that by the principal axis angle calculation method based on image moments, the main extension direction of the rib in the image can be accurately determined, which has good noise resistance and stability, and compared with the direct edge detection method, it can more reliably process the rib shape change and unclear boundary problem caused by fracture; the threshold selection in the binaryzation processing ensures that stable binaryzation effect can be obtained under different CT scanning conditions, providing a reliable foundation for subsequent moment calculation.

[0044] Further, the rotation correction and scaling to a uniform size of the cropped image according to the principal axis angle includes rotating the cropped image by the principal axis angle, so that the rib is basically in a vertical state in the slice image. The rotated image is scaled in proportion, so that the long side size is 100 pixels, and the short side is filled to 100 pixels around 0 value in symmetry, to obtain an input image with a uniform size of 100x100 pixels, forming a multi-instance slice image.

[0045] By uniformly adjusting the rib to a vertical state, the problem of inconsistent direction of ribs in different positions in the CT image is eliminated, so that the network can perform feature learning under a standardized spatial configuration, improving the consistency and comparability of feature extraction; uniform size processing not only meets the format requirements of network input, but more importantly, by maintaining the aspect ratio of the image and using 0 value padding, deformation distortion caused by direct stretching is avoided, ensuring the authenticity of the rib morphology feature, laying a solid data foundation for subsequent multi-scale feature extraction, so that CT images of different sources and different scanning conditions can be processed under a unified coordinate system and scale, improving the generalization ability and prediction accuracy of the model.

[0046] S2: inputting the multi-instance slice image into a multi-scale feature extraction network to extract four multi-instance features of different scales respectively.

[0047] In this embodiment, as Figure 3 The diagram shows the overall network structure of this invention, including the stem layer, four feature extraction stages, feature alignment module, packet fusion module, and the connection relationships of the packet regressor. Figure 3 The data flow and information transmission paths between modules are clearly marked, intuitively demonstrating the complete network architecture from multi-instance slice image input to fracture time prediction output, reflecting the design ideas of multi-scale feature extraction and multi-instance fusion. In particular, by integrating multiple preprocessed multi-instance slice images of the same fracture site into a complete data packet input network, the rich feature information of the lesion in different image sections can be fully utilized. First, the original features of each slice image are fully extracted and preliminarily analyzed, and then the features of all slice images are fused to achieve comprehensive judgment.

[0048] Specifically, the multi-instance slice image is input into a multi-scale feature extraction network to extract multi-instance features at four different scales. This includes preliminary feature extraction and size compression of the multi-instance slice image through a stem layer. The image output from the stem layer is then sequentially input into the first, second, third, and fourth feature extraction stages. Each feature extraction stage processes the image output from the stem layer through a pooling attention module, outputting multi-instance features at the corresponding scale, resulting in four multi-instance features at different scales.

[0049] Preliminary feature extraction and size compression of the multi-instance slice images through the stem layer includes, assuming the input image... ,in The number of cases refers to the number of cases for different fracture lesions. The value is variable and is determined based on the number of slices showing the lesion location in the current CT image. Input image After including The stem layer, which incorporates convolution and downsampling, reduces the feature map size and alleviates computational burden. As the initial processing module of the network, the stem layer performs appropriate size compression and preliminary feature abstraction on the input image in the early stages of feature extraction, laying the foundation for subsequent multi-scale processing. Furthermore, the choice of convolution kernel size fully considers the spatial characteristics of the rib structure. The convolutional kernel can effectively capture local features of rib edges and textures. In addition, the feature map dimensionality is compressed through downsampling, which improves the computational efficiency of subsequent processing while maintaining key information, laying the foundation for clinical application scenarios with high real-time requirements.

[0050] The image output by the stem layer is sequentially input into the first feature extraction stage, the second feature extraction stage, the third feature extraction stage, and the fourth feature extraction stage, that is, the image output by the stem layer is sequentially subjected to the feature extraction operations of the four stages to obtain four different scale features. Specifically, the image output by the stem layer is first processed by the pooling attention module to obtain first scale multi-instance features , and then sequentially subjected to downsampling and the pooling attention module to obtain second scale multi-instance features , third scale multi-instance features , and fourth scale temporary features . The fourth scale temporary features are subjected to average pooling and flattening processing to obtain fourth scale multi-instance features .

[0051] It should be noted that, considering that the diversity of fracture damage degree can cause large differences in feature scale, the network needs to generate features of different scales during feature extraction, and each scale contains multiple instance features. The embodiment can mine multi-level feature information related to fractures layer by layer from fine granularity to coarse granularity through four ordered feature extraction operations, and each stage focuses on feature extraction of a specific scale, avoiding important information that may be missed by single scale features.

[0052] Further, in order to fully utilize the characteristics that the fracture site usually has high-density features, focus on the region of interest, and suppress irrelevant noise, the embodiment introduces a pooling attention module with channel attention and spatial attention mechanisms in the multi-scale feature extraction network to guide the network to focus on the high-density area of the fracture, effectively improving the feature representation capability. As shown in Figure 4 , which is a structural diagram of the pooling attention module, including channel attention and spatial attention, the diagram shows the complete process of input features passing through a convolution layer, a linear layer, a global pooling operation, an excitation operation, and attention weight calculation, as well as the synergistic mechanism of channel attention and spatial attention, highlighting the technical features of the pooling attention module in focusing on the high-density area of the fracture.

[0053] The channel attention includes sequentially passing the input feature F0 through a convolution layer, a linear layer, a Gaussian Error Linear Units (GELU) activation, and a linear layer to obtain an intermediate feature F1; performing global average pooling and global maximum pooling operations on the intermediate feature F1 in the spatial dimension to obtain channel-wise average features and maximum features . The average features and the maximum features The shared weight excitation operation is performed to fully excavate the dependency between the channels and obtain average features and maximum features of the excitation operation result , the excitation operation comprising: compressing the channel dimension by means of a linear dimension reduction layer with a reduction ratio = 16, passing through a nonlinear activation function ReLU, and then recovering to the original channel dimension of the input feature through a linear dimension increase layer. The excitation operation results of the average features and the maximum features are added element by element, and are calculated through a Sigmoid activation function to obtain channel attention weights . The channel attention weights and the intermediate features F1 are multiplied channel by channel to obtain channel attention output features F2.

[0054] Specifically, the specific formula of the channel attention weights is as follows:

[0055] wherein, is the channel attention weight; and are the weight vectors of the dimension reduction layer and the dimension increase layer, respectively.

[0056] Considering that the fracture injury site is often accompanied by callus formation, which will increase the bone density of the corresponding area, and the callus morphology is the key basis for judging the fracture time, therefore, in order to effectively focus on the callus area detail features, so that the model can more accurately capture the bone density change and callus morphology, and strengthen the fracture time discrimination ability, the spatial attention is used to further process the channel attention output features F2.

[0057] The spatial attention comprises: adopting average pooling processing on the multi-instance slice image in the spatial dimension, adjusting the multi-instance slice image to the same spatial scale as the channel attention output features F2, and then normalizing by using a softmax function to generate spatial attention weights . The spatial attention weights and the channel attention output features F2 are multiplied pixel by pixel to obtain the final output features of the pooling attention module.

[0058] To solve the problem that the rib fracture area is small and the boundary is fuzzy relative to the whole CT image volume, and the traditional method is difficult to accurately locate and extract the fracture-related features, a multi-scale feature extraction network combined with a pooling attention mechanism is proposed. By introducing a combination mechanism of channel attention and spatial attention, the formation area of bone callus and the bone density change characteristics can be identified and focused. Through four ordered stages of feature extraction operations, from fine-grained bone callus microstructure to coarse-grained bone fracture overall profile, a complete feature hierarchy from micro to macro is established. The pooling attention module uses the synergistic effect of global average pooling and maximum pooling to retain the global statistical information of the features and highlight the detailed features of the local key regions. Based on the physiological characteristics that bone callus formation in the fracture injury site will increase the bone density of the region, the spatial attention mechanism can accurately locate the corresponding high-density area and effectively suppress the interference of irrelevant background information. This makes the network accurately capture multi-level features related to fracture time while maintaining computational efficiency, providing high-quality feature representation for subsequent time prediction.

[0059] S3: performing feature alignment processing on the multi-instance features of the first three scales to obtain aligned multi-instance features, and generating instance evidence values based on the multi-instance features of the fourth scale, and calculating the contribution weight of the instance according to the instance evidence values.

[0060] To solve the problem of spatial consistency and contribution difference of different instance features in multi-instance learning, on the one hand, the multi-instance features of the first three scales are aligned to solve the possible feature shift problem between instances; on the other hand, instance evidence values are generated based on the multi-instance features of the fourth scale to provide reliable weight basis for subsequent feature fusion.

[0061] Specifically, the feature alignment processing on the multi-instance features of the first three scales to obtain the aligned multi-instance features includes inputting the multi-instance features of the first three scales into a feature alignment module respectively. The multi-instance features of the first three scales are processed by feature offset learning and alignment based on the offset through deformable convolution. The features after deformable convolution are sequentially processed by layer normalization and linear transformation to obtain the aligned multi-instance features.

[0062] In the same fracture lesion of multiple slice images, the fracture site will shift slightly due to the difference between slice images and the uncertainty in the manual framing process, so it is necessary to align the features of different slice images. By inputting the multi-instance features of the first three scales into the feature alignment module for inter-instance alignment, the aligned multi-instance features are obtained. 、 and are input into the feature alignment module for inter-instance alignment, respectively, to obtain the aligned multi-instance features, wherein Figure 5The feature alignment module is shown in the structural diagram, and is composed of a deformable convolution, a layer normalization, a linear layer, a ReLU activation function and a linear layer, for realizing accurate alignment between features, and obtaining multi-instance features after alignment.

[0063] The feature offset learning and offset-based alignment of the multi-instance features of the first three scales by the deformable convolution are core links of feature alignment; the deformable convolution can adjust the sampling position of the convolution kernel according to the spatial distribution characteristics of the input features, so as to learn the spatial offset mode between different instances; compared with the traditional fixed convolution kernel, the deformable convolution can better adapt to the position change of the fracture part in different slices, realize implicit alignment between features, and effectively eliminate the spatial inconsistency problem caused by the image acquisition angle and manual frame selection error factors through accurate modeling of the feature offset.

[0064] Secondly, in the layer normalization and linear transformation processing of the features after the deformable convolution, the layer normalization operation ensures the consistency of different instance features in the numerical distribution, avoiding the influence of amplitude difference on the subsequent processing effect. Linear transformation further adjusts the representation space of the features, so that the aligned features are more suitable for subsequent fusion operation, ensuring the accuracy and stability of feature alignment, and laying a foundation for high-quality feature fusion.

[0065] Further, the multi-instance feature of the fourth scale generates an instance evidence value, and the contribution weight of the instance is calculated according to the instance evidence value, which includes inputting the multi-instance feature of the fourth scale into a multi-layer perception machine to output a distribution parameter. According to the evidence theory, the evidence value corresponding to each instance is calculated according to the distribution parameter. The distribution parameter is explicitly constrained by an evidence loss function. The evidence value corresponding to each instance is normalized to obtain the contribution weight of each instance.

[0066] Inputting the multi-instance feature of the fourth scale into a multi-layer perception machine to output a distribution parameter means that for a regression task facing a continuous label, the fourth scale multi-instance feature is input into a multi-layer perception machine for processing to output a distribution parameter, which provides a theoretical basis for subsequent evidence value calculation. Specifically, assuming that the prediction target variable y is independently and identically distributed and subject to a Gaussian distribution , the mean is subject to a Gaussian distribution , the variance is subject to an inverse gamma distribution , where is the mean parameter of the mean distribution, is the scale parameter of the mean distribution, is the shape parameter of the variance distribution, is a scale parameter of the variance distribution. The Gaussian distribution is predicted by the multi-layer perception and inverse gamma distribution Four groups of parameters in the inverse gamma distribution, that is 、 、 and .

[0067] Based on the evidence theory, the evidence value corresponding to each instance is calculated according to the distribution parameter , and the specific formula is as follows:

[0068] wherein, is the evidence value; is a scale parameter of the mean distribution; is a shape parameter of the variance distribution.

[0069] It should be noted that the size of the evidence value directly reflects the credibility of the corresponding instance, and then reflects the contribution size of each slice image in the model decision process; the larger the evidence value, the stronger the support of the corresponding instance to the final prediction result, and a higher weight should be given in the feature fusion process.

[0070] The explicit constraint of the distribution parameter by the evidence loss function means that the evidence value is explicitly constrained by the corresponding loss function, and then the feasible contribution of each instance to the regression target is reflected, avoiding the randomness and instability of weight allocation, which is a key step to ensure the reliability of the evidence value, wherein the distribution parameter is optimized by minimizing the evidence loss function, and the specific formula of the evidence loss function is as follows:

[0071] wherein, is the evidence loss function; is the Gamma function.

[0072] The evidence value corresponding to each instance is normalized to obtain the contribution weight of each instance, that is, the evidence value is normalized by the softmax to obtain the contribution weight of each instance , ensuring that the sum of all weights is 1, which meets the basic requirement of probability distribution, wherein by performing the normalization processing, not only the rationality of the weight is ensured, but also the importance comparison between different instances is more intuitive and interpretable, which provides a reliable quantitative basis for subsequent feature fusion.

[0073] To solve the problem of feature space inconsistency caused by scanning accuracy difference and artificial labeling uncertainty in multiple slice images of the same fracture lesion, a feature alignment module based on deformable convolution and an instance weight calculation method based on evidence theory are proposed. The feature offset learning through deformable convolution can adjust the sampling position of the convolution kernel and learn the spatial offset mode between different instances, which has stronger adaptability than traditional fixed convolution kernel. It can effectively eliminate the spatial inconsistency caused by factors such as image acquisition angle and artificial labeling error, and realize accurate alignment at the feature level. At the same time, the weight calculation based on evidence theory introduces statistical principles into the multi-instance learning framework by predicting four groups of parameters of Gaussian distribution and inverse gamma distribution, and calculates the evidence value of each instance to quantify the reliability. Compared with traditional average pooling or simple attention mechanism, the present application can more accurately reflect the actual contribution of each slice image in model decision making, and through the constraint of evidence loss function, it ensures the stability and interpretability of weight distribution, providing a reliable quantitative basis for subsequent feature fusion.

[0074] S4: According to the contribution weight, the aligned multi-instance features and the multi-instance features of the fourth scale are respectively fused to obtain four package features of different scales.

[0075] In this embodiment, the obtained contribution weight is used to differentially process and fuse different instance features. The aligned multi-instance features of the first three scales 、 and are respectively aligned between instances through the feature alignment module, and then input into the package fusion module and the pooling attention module to obtain the package features of the first three scales. The operation of the pooling attention module is to convert the spatial dimension of the feature to 1 after convolution of the input feature and the convolution kernel with the same size as the input feature. In addition, for the multi-instance features of the fourth scale , on the one hand, it is directly input into the package fusion module to obtain the fourth scale package feature, and on the other hand, it is input into the multi-layer perception to calculate and generate the evidence value , and then the softmax calculation is performed to obtain the instance contribution weight . The instance contribution weight is input into the package fusion module of each scale to fuse the instance features.

[0076] Specifically, according to the contribution weight, the aligned multi-instance feature and the multi-instance feature of the fourth scale are respectively subjected to feature fusion to obtain four package features of different scales, including sorting the contribution weights according to the size, determining the instance with the highest evidence value as the main instance, and the remaining instances as reference instances. The main instance feature and the reference instance feature are subjected to deep interaction integration through an instance feature interaction module. The main instance feature and the interaction-integrated reference instance feature are weighted and accumulated according to the contribution weights. The steps from sorting the contribution weights according to the size to the weighted accumulation are performed on the aligned multi-instance feature and the multi-instance feature of the fourth scale respectively to obtain four package features of different scales.

[0077] The sorting of the contribution weights according to the size includes, for the evidence value of each instance, sorting each value from large to small to obtain . The contribution weight of each instance after softmax normalization . The instance with the highest evidence value, i.e. , is defined as the main instance, which plays a leading role in the fracture prediction result; the remaining corresponding instances are defined as reference instances, which are used to supplement the main instance information.

[0078] Further, as Figure 6 is a structural diagram of the package fusion module, including the division of the main instance and the reference instance, the instance feature interaction process and the weighted fusion operation, which is used to illustrate how the main instance and the reference instance are subjected to deep interaction through the attention mechanism, and finally realize intelligent feature fusion based on credibility.

[0079] Specifically, when the main instance feature and the reference instance feature are subjected to deep interaction integration through the instance feature interaction module, the main instance feature is represented as , and the reference instance feature is represented as (k is the reference instance number). Since the main instance feature itself has the most core dominant feature, it does not need additional interaction operation. However, for the reference instance feature, in order to effectively fuse the supplementary information contained in the reference instance feature with the main instance, the reference instance feature needs to be input into the instance feature interaction module together with the main instance feature, so as to mine the feature correlation between instances.

[0080] Secondly, as Figure 7The example feature interaction module is shown, including the introduction of position encoding, the generation process of query vector Q, key vector K, and value vector V, and the specific operation of attention weight calculation and feature fusion. In the interactive attention operation link, the input is the main instance feature and a reference instance feature, and the learnable position encoding is introduced, which is added to the main instance and the reference instance feature respectively, so that the model can pay attention to the content feature and the spatial layout feature at the same time when interacting. The main instance feature generates the query vector Q through the linear layer, and the reference instance feature generates the key vector K and the value vector V through two independent linear layers respectively. The similarity between the query vector Q and the key vector K is calculated through the attention operation, and then the value vector V is weighted and fused according to the similarity to obtain the fused feature, realizing the deep interactive integration of the main instance feature and the reference instance feature.

[0081] The specific formula of the deep interactive integration of the main instance feature and the reference instance feature is as follows: ; Wherein, FF is the instance feature interaction operation; d is the dimension of the key vector.

[0082] It should be noted that, in order to ensure the stability and consistency of the feature interaction process, the instance feature interaction module uniformly adopts shared weights for all instances, so that the reference instance can adjust the corresponding feature representation based on the relevance degree with the main instance, thereby realizing more accurate and targeted feature supplement.

[0083] Further, in order to better retain the discriminative information of the key instance and reasonably utilize the supplement value of the reference instance, the quality and prediction performance of the fused feature are improved, the main instance feature and the interactive reference instance feature are weighted and accumulated according to the instance contribution weight, and finally the bag feature is obtained, and the specific formula is as follows: ; Wherein, is the bag feature; FF is the instance feature interaction operation.

[0084] For the aligned multi-instance features at the first three scales, the processes are performed according to the aforementioned steps of contribution weight sorting, principal parameter instance determination, feature interaction, and weighted accumulation to obtain the corresponding scale's bag features. For the fourth-scale multi-instance features, the processing measures are the same as for the aligned multi-instance features at the first three scales. The difference is that since the spatial dimension of the fourth-scale multi-instance features has been compressed to 1, pooling is not required after fusion. Through this unified processing framework, multi-instance features at four different scales can achieve evidence-based fusion while maintaining their respective scale characteristics, ultimately resulting in four bag features at different scales. This retains the advantages of multi-scale feature extraction and, through an evidence-theoretic-guided fusion strategy, ensures the accuracy and discriminative power of feature representation, providing high-quality feature input for subsequent fracture time prediction.

[0085] To address the issue that simple averaging fusion in existing multi-instance learning methods dilutes key information in continuous label regression tasks, this paper proposes a new approach. Instances are divided into master instances and reference instances based on their evidence values. This ensures that the most discriminative slice plays a dominant role in the fusion process, avoiding the dilution of key information that might result from simple averaging fusion. An attention mechanism enables deep interactive integration between master and reference instances, allowing reference instances to adjust their feature representations based on their relevance to the master instances, thus providing more accurate and targeted supplementary information. During the weighted accumulation process, the contribution of each instance is proportional to its evidence value, reflecting the credibility-based feature fusion concept. To ensure the stability and consistency of the feature interaction process, the instance feature interaction module adopts a shared weight mechanism for all instances. This ensures the dominant position of key instances while fully utilizing the supplementary value of other instances. Compared to traditional equal-weighted averaging or max-pooling methods, this approach better preserves the discriminative information of key instances, improving the quality of fused features and predictive performance.

[0086] S5: After splicing the bag features of the four different scales, input them into the bag regressor to obtain the prediction result of the rib fracture time.

[0087] Specifically, the method of concatenating the four different scales of package features and inputting them into the package regressor to obtain the prediction result of rib fracture time includes concatenating the four different scales of package features in the feature dimension to form the final feature of fracture time regression. By concatenating the four different scales of package features in the feature dimension, multi-level feature information from fine-grained to coarse-grained can be fully integrated, providing comprehensive feature input for subsequent regression prediction.

[0088] The final fusion feature is input into a package regressor for processing, wherein the package regressor is composed of a multilayer perceptron and a Sigmoid activation function, the mapping relationship between the feature and the fracture time is learned through nonlinear transformation of the multilayer perceptron, and the Sigmoid activation function ensures that the fracture time prediction result is in the interval of 0 to 1.

[0089] The fracture time prediction result is multiplied by the actual fracture day maximum value to obtain the predicted fracture day. In an embodiment, the actual fracture day maximum value is 90, and the normalized prediction value is converted into a day representation with actual clinical significance through linear transformation to obtain the final prediction result of the rib fracture time.

[0090] It should be noted that the mean absolute error loss is used as the loss function in the training process of the rib fracture time prediction described above. The mean absolute error loss is a commonly used loss function in regression tasks, which measures the model performance by calculating the average absolute error between the predicted value and the true value. The calculation process is simple and efficient, and has strong robustness to abnormal values, and will not cause the error to be excessively enlarged due to extreme values; the specific formula of the mean absolute error loss is as follows:

[0091] Wherein, N is the number of training samples, is the true value of the t-th sample, is the prediction value of the t-th sample.

[0092] The total loss function is the sum of the mean absolute error loss and the evidence loss .

[0093] Wherein, is the total loss function.

[0094] To solve the problem that single scale feature cannot cover all the judgment basis for fracture time judgment which needs to integrate multi-level information, four different scale bag features are spliced in feature dimension to form a complete feature representation containing from macro fracture contour to micro callus details, avoiding information loss caused by simple feature addition or average fusion; the bag regressor adopts the combination design of multilayer perceptron and Sigmoid activation function, learns the mapping relationship between fused features and fracture time through deep nonlinear transformation, has stronger expression ability than traditional linear regression method, and can capture the nonlinear change rule in fracture healing process; the Sigmoid activation function ensures that the output value is limited in the interval of zero to one, which is convenient for subsequent conversion to actual days; the joint optimization strategy of mean absolute error loss and evidence loss is adopted in the model training process, which ensures the optimization of prediction accuracy and the rationality of instance weight distribution.

[0095] In another aspect, the embodiment also provides a rib fracture time prediction system based on multi-instance learning, which comprises: An image preprocessing module acquires CT image data of rib fracture, pre-processes the CT image data, and obtains multi-instance slice images.

[0096] A feature extraction module inputs the multi-instance slice images into a multi-scale feature extraction network to extract four different scale multi-instance features respectively.

[0097] A feature alignment module performs feature alignment processing on the multi-instance features of the first three scales to obtain aligned multi-instance features, generates instance evidence values based on the multi-instance features of the fourth scale, and calculates the contribution weights of instances according to the instance evidence values.

[0098] A bag fusion module performs feature fusion on the aligned multi-instance features and the multi-instance features of the fourth scale respectively according to the contribution weights to obtain four different scale bag features.

[0099] A time prediction module inputs the four different scale bag features after splicing into a bag regressor to obtain the prediction result of rib fracture time.

[0100] If the above functions are realized in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or parts of the present application that make contributions to the prior art or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0101] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a list of executable instructions for implementing logic functions, which can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus or device, such as a computer-based system, a system including a processor or other system that can fetch instructions from an instruction execution system, apparatus or device and execute the instructions, or in conjunction with these instruction execution systems, apparatus or devices. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transport programs for use by or in connection with an instruction execution system, apparatus or device, or in conjunction with these instruction execution systems, apparatus or devices.

[0102] More specific examples (non-exhaustive list) of the computer-readable medium include the following: an electrical connection having one or more wires (electrical devices), a portable computer diskette (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CD ROM). In addition, the computer readable medium can even be paper or other suitable medium on which the program can be printed, because the program can be obtained electronically, for example, by optical scanning of the paper or other medium, followed by editing, interpreting or otherwise processing, if necessary, in other suitable ways, to be electronically obtained and then stored in the computer memory.

[0103] It should be understood that portions of the application can be implemented in hardware, software, firmware, or combinations thereof. In the embodiments described above, the various steps or methods can be implemented, in part, or in whole, by software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, and in another embodiment, any of the following techniques can be used to implement the hardware: discrete logic circuitry having logic gates for implementing logic functions upon data signals, application specific integrated circuits having logic gates, field programmable gate arrays (FPGAs), and so on, or combinations thereof.

[0104] Example 2, which is an embodiment of the present application, provides a rib fracture time prediction method based on multi-instance learning. In order to verify the beneficial effects of the present application, economic benefit calculation and simulation experiments are used for scientific demonstration.

[0105] In order to verify the effectiveness of the rib fracture time prediction method based on multi-instance learning proposed in the present application, the private data set used in the experiment contains 350 three-dimensional chest CT image data from multiple centers, and is equipped with corresponding injury time information; the rib region is labeled by forensic experts, and a total of 1864 rib fracture lesions are involved; the samples are divided into training set, validation set and test set in the ratio of 8:1:1, and at the same time, it is strictly ensured that all image data of the same patient only appears in the same data subset, effectively avoiding the risk of data leakage.

[0106] First, the mean absolute error MAE and the coefficient of determination R 2 As an evaluation index of the regression task, the prediction performance of the model is comprehensively measured, and the specific formula is as follows:

[0107]

[0108]

[0109]

[0110] Wherein, M is the number of test samples; is the true value of the tth sample; is the predicted value of the tth sample; is the mean of the true label; SST is the total sum of squares; SSE is the residual sum of squares.

[0111] Secondly, in order to verify the effectiveness of each key module of the application, the application carries out corresponding ablation experiments on the pooling attention, multi-scale feature and multi-instance evidence fusion method; in addition, in the comparative experiment, the method of the application is compared with other networks based on multi-instance learning, including Mean-pooling, Max-pooling, ABMIL, TransMIL and DSMIL, and the effectiveness of the method proposed in the application is verified; wherein the results of the ablation experiment are shown in Table 1, and the results of the comparative experiment are shown in Table 2.

[0112] Table 1 Ablation experiment results of multi-instance rib fracture time prediction

[0113] Table 2 Comparative experiment results of multi-instance rib fracture time prediction

[0114] As can be seen from Table 1, compared with the baseline network (ABMIL), the method of the application has improved in two key indicators, that is, the mean absolute error MAE is reduced from 9.59 days to 8.50 days, reduced by 1.09 days; the determination coefficient R2 is improved from 0.640 to 0.714, improved by 0.074, and the optimization of these indicators fully embodies the performance advantage of the model in the fracture time prediction task; in addition, as can be seen from Table 2, the method of the application has improved in each evaluation indicator compared with the existing multi-instance learning method, fully verifying that the multi-instance learning network proposed in the application can better realize the accurate prediction of the fracture time.

[0115] It should be noted that the above embodiments are only used to illustrate the technical solutions of the application and not to limit it, although the application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the application can be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the application, and they should be covered in the scope of the claims of the application.

Claims

1. A method for predicting rib fracture time based on multi-instance learning, characterized in that, include: CT image data of rib fractures are acquired, and the CT image data is preprocessed to obtain multi-instance slice images; The multi-instance slice image is input into a multi-scale feature extraction network to extract multi-instance features at four different scales. Feature alignment is performed on the multi-instance features of the first three scales to obtain aligned multi-instance features, and instance evidence values ​​are generated based on the multi-instance features of the fourth scale. The contribution weight of the instance is calculated based on the instance evidence values. Based on the contribution weight, the aligned multi-instance features and the fourth-scale multi-instance features are respectively fused to obtain four bag features at different scales. The packet features at the four different scales are concatenated and then input into the packet regressor to obtain the predicted time of rib fracture.

2. The rib fracture time prediction method based on multi-instance learning as described in claim 1, characterized in that: Preprocessing the CT image data to obtain multi-instance slice images includes marking the rib region in the CT image data, determining the minimum bounding rectangle containing the rib region, and cropping it. The cropped image is binarized to obtain a binary image. The moments of each order are calculated and the principal axis angles are determined based on the binary image. The cropped image is rotated and scaled to a uniform size based on the principal axis angle to obtain a multi-instance slice image.

3. The rib fracture time prediction method based on multi-instance learning as described in claim 2, characterized in that: The multi-instance slice image is input into a multi-scale feature extraction network to extract multi-instance features at four different scales. Preliminary feature extraction and size compression are performed on the multi-instance slice images using a stem layer, wherein the stem layer includes... Convolutional layers and downsampling; The image output from the stem layer is sequentially input into the first feature extraction stage, the second feature extraction stage, the third feature extraction stage, and the fourth feature extraction stage. Each feature extraction stage processes the image output from the stem layer through a pooling attention module, outputting multi-instance features at the corresponding scale, resulting in four multi-instance features at different scales. The pooling attention module includes channel attention and spatial attention.

4. The rib fracture time prediction method based on multi-instance learning as described in claim 3, characterized in that: The step of performing feature alignment processing on the multi-instance features of the first three scales to obtain aligned multi-instance features includes inputting the multi-instance features of the first three scales into the feature alignment module respectively. Feature offset learning and offset-based alignment are performed on the multi-instance features of the first three scales using deformable convolution. The features after deformable convolution are sequentially processed by layer normalization and linear transformation to obtain aligned multi-instance features.

5. The rib fracture time prediction method based on multi-instance learning as described in claim 4, characterized in that: The generation of instance evidence values ​​based on the fourth-scale multi-instance features, and the calculation of the instance contribution weights based on the instance evidence values, include inputting the fourth-scale multi-instance features into a multilayer perceptron and outputting distribution parameters. Based on evidence theory, the evidence value corresponding to each instance is calculated according to the distribution parameters; The distribution parameters are explicitly constrained by the evidence loss function; The evidence value corresponding to each instance is normalized to obtain the contribution weight of each instance.

6. The rib fracture time prediction method based on multi-instance learning as described in claim 5, characterized in that: Based on the contribution weights, the aligned multi-instance features and the fourth-scale multi-instance features are respectively fused to obtain bag features at four different scales, including: The contribution weights are sorted by size, and the instance with the highest evidence value is determined as the main instance, while the remaining instances are reference instances. The main instance features and reference instance features are deeply integrated and interacted with each other through the instance feature interaction module; The main instance features and the reference instance features after interaction integration are weighted and accumulated according to the contribution weights. The aligned multi-instance features and the fourth-scale multi-instance features are processed by performing a step from sorting the contribution weights by size to a weighted accumulation step, resulting in four different scales of bag features.

7. The rib fracture time prediction method based on multi-instance learning as described in claim 6, characterized in that: The four different scales of bag features are concatenated and then input into the bag regressor to obtain the prediction result of rib fracture time. This includes concatenating the four different scales of bag features along the feature dimension to form the final fused feature. The final fused features are input into a bag regressor for processing, wherein the bag regressor consists of a multilayer perceptron and an activation function, and outputs a fracture time prediction value within a preset range. The predicted fracture time is converted into the actual number of days after the fracture to obtain the predicted rib fracture time.

8. A rib fracture time prediction system based on multi-instance learning, based on the rib fracture time prediction method based on multi-instance learning according to any one of claims 1-7, characterized in that: The image preprocessing module acquires CT image data of rib fractures, preprocesses the CT image data, and obtains multi-instance slice images. The feature extraction module inputs the multi-instance slice image into a multi-scale feature extraction network to extract multi-instance features at four different scales. The feature alignment module performs feature alignment processing on the multi-instance features of the first three scales to obtain aligned multi-instance features, and generates instance evidence values ​​based on the multi-instance features of the fourth scale, and calculates the contribution weight of the instance based on the instance evidence values. The packet fusion module fuses the aligned multi-instance features and the fourth-scale multi-instance features according to the contribution weights to obtain packet features at four different scales. The time prediction module concatenates the bag features at four different scales and inputs them into the bag regressor to obtain the prediction result of the rib fracture time.

9. A computer device, comprising: A memory and a processor; the memory stores a computer program, characterized in that: when the processor executes the computer program, it implements the steps of the method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1-7.

Images