Automatic centrum fracture recognition and analysis system based on artificial intelligence

Abstract

The invention discloses a centrum fracture automatic identification and analysis system based on artificial intelligence, and relates to the technical field of image identification, and the centrum fracture automatic identification and analysis system comprises the following steps: carrying out regulation and control processing operation on a tomography image, and carrying out answer marking on the basis of the processed tomography image; constructing an automatic identification network structure, training the automatic identification network structure by using the tomography image and the answer label, and identifying fracture information by using the trained automatic identification network structure; the fracture severity is analyzed according to the fracture information, the fracture influence degree is evaluated according to the severity analysis result, and the severity analysis result and the fracture influence degree are transmitted to the mobile terminal. The centrum fracture automatic identification and analysis system based on artificial intelligence is provided through the artificial intelligence technology, the size, the position and the type of the fracture are analyzed through the artificial intelligence algorithm to judge the severity, and the possible influence on a patient is evaluated.

Description

Technical Field

[0001] The present invention relates to the field of image recognition technology, and in particular to an automatic recognition and analysis system for vertebral fractures based on artificial intelligence. Background Art

[0002] With the popularization of informatization, especially with the gradual development of artificial intelligence technology, people are using more and more scientific and technological products. Especially in the medical industry, how to use artificial intelligence technology to improve the level of medical services is particularly important, among which artificial intelligence technology based on deep learning has received more and more attention and is widely used.

[0003] Vertebral compression fractures are one of the most common types of osteoporotic fractures. Conventional preliminary diagnosis relies on taking anteroposterior and lateral spinal X-rays and visual diagnosis by radiologists. The accuracy is closely related to the experience of radiologists. Doctors mainly make diagnoses based on the Genant visual semi-quantitative judgment method. However, due to the accumulation of gas and feces in the abdomen of elderly patients, doctors are often unable to accurately judge the degree of fracture compression, which in turn affects the formulation of subsequent treatment plans. In existing technologies, most input data is normalized or binarized before subsequent analysis, which brings additional interference to subsequent analysis, and most solutions take more than a few minutes to run.

[0004] For example, Chinese patent CN114937502A discloses an osteoporotic vertebral compression fracture assessment method and system based on deep learning, which uses a detection model and a segmentation model for analysis, and binarizes the input image before analysis. Since the captured CT images are not fixed, using a single threshold for binarization will interfere with subsequent analysis, and the two models are processed serially, which reduces the computing performance.

[0005] At the same time, the paper "Automatic L3 slice detection in 3D CT images using fully-convolutional networks" opened up the CT multi-view dataset, which proposed a Unet-based deep learning solution for 3D image recognition and analysis. Although the fully convolutional network can effectively process medical images, such as CT images.

[0006] However, 3D image processing itself increases the computational cost, and the conversion of 3D to 2D images will cause image information loss. In addition, with the influence of angle, light and other environmental factors, interference between image context information will be caused. Therefore, although the full convolutional network can adapt to different input image sizes, the classification accuracy and positioning accuracy cannot be achieved at the same time. When the receptive field is selected to be relatively large, the corresponding pooling layer will reduce the dimensionality multiple.

[0007] Currently, no effective solution has been proposed for the problems in the related technologies. Summary of the invention

[0008] In view of the problems in the related technology, the present invention proposes an automatic identification and analysis system for vertebral fractures based on artificial intelligence to overcome the above-mentioned technical problems existing in the existing related technology.

[0009] To this end, the specific technical solution adopted by the present invention is as follows: The automatic identification and analysis system of vertebral fracture based on artificial intelligence includes: A scanning image processing and annotation unit, used for performing a control and processing operation on the tomographic scanning image, and annotating the answer based on the tomographic scanning image after the processing; Vertebral fracture automatic recognition unit, used to construct an automatic recognition network structure, train the automatic recognition network structure using tomographic images and answer annotations, and recognize fracture information using the trained automatic recognition network structure; The fracture information recognition and judgment unit is used to analyze the severity of the fracture based on the fracture information, use the severity analysis results to evaluate the impact of the fracture, and transmit the severity analysis results and the impact of the fracture to the mobile terminal.

[0010] Preferably, the scanned image processing and annotation unit includes: An image rotation enhancement module is used to perform rotation processing on the tomographic image using image rotation technology and implement image enhancement operations after the rotation is completed; An image affine scaling module is used to perform affine transformation processing on the tomographic image and to scale the tomographic image using bilinear interpolation technology after the affine transformation; The scanned image annotation module is used to annotate the answers of the processed tomographic scan images according to the input form of the deep learning network.

[0011] Preferably, performing rotation processing on the tomographic image using the image rotation technology and performing image enhancement operation after the rotation is completed includes: The tomographic image is rotated counterclockwise according to a predefined rotation point using an image rotation technology, and the rotation stops after reaching a preset angle; Combining the initial coordinate position of the tomographic image with the predefined rotation point to obtain the coordinate position of the rotated tomographic image; evaluating the image quality of the rotated tomographic image and fusing the evaluation result with a noise generation technique to add noise to the tomographic image; The noise-added tomographic image is subjected to inverse transformation to complete the noise removal operation, thereby obtaining an enhanced tomographic image.

[0012] Preferably, performing affine transformation processing on the tomographic image, and scaling the tomographic image using bilinear interpolation technology after the affine transformation includes: After determining the type and parameters of the affine transformation based on the tomographic image, a transformation matrix is ​​constructed, and a mapping position of the tomographic image is defined by using the transformation matrix; Performing affine transformation on the tomographic image using the transformation matrix relationship to obtain a mapping coordinate matrix of the changed tomographic image; Based on the bilinear interpolation technology and the tomographic image, the tomographic image is enlarged or scaled in the longitudinal and transverse directions.

[0013] Preferably, performing answer annotation processing on the processed tomographic image according to the input form of the deep learning network includes: Based on the rectangular annotation technology, the main area frame of the vertebra in the tomographic image is annotated to obtain the complete area of ​​the vertebra, and the pixel-level annotation is used to perform secondary annotation in the complete area of ​​the vertebra to obtain the vertebral sub-area; The vertebral sub-regions are digitally labeled in the order of labeling, and after the labeling is completed, the classification information of the vertebral sub-regions is calibrated to determine the fracture status of the vertebral body; Obtain personal information, injury mechanism and fracture history of the person corresponding to the tomographic image based on the image acquisition information, and combine the corresponding tomographic image with expert annotation technology to perform fracture status assessment and annotation; The personal information, injury mechanism, fracture history and fracture status assessment annotation results of the corresponding person are integrated into the answer annotation text.

[0014] Preferably, the vertebral fracture automatic identification unit comprises: The network structure building module is used to build a bimodal network and a large text semantic model structure, and automatically identify the network structure based on the construction results; Structural parameter configuration module, used to configure the hyper parameters of automatic identification network structure and training files; The recognition network structure training module is used to input the tomography image and the answer annotation into the automatic recognition network structure to perform network training and verify the evaluation accuracy of the output vertebral area fracture information; The network structure adjustment module is used to adjust the automatic recognition network structure based on the evaluation accuracy results.

[0015] Preferably, constructing a bimodal network and a text semantic large model structure, and obtaining an automatic recognition network structure based on the construction result includes: The deep convolutional neural network model is selected as the basic network structure for feature extraction, the scale feature layer is set in combination with the pyramid structure, and convolution and deconvolution operations are performed on each scale feature layer; After the convolution is completed, a feature fusion operation is performed, and the branch functions of each scale feature layer are defined to obtain a bimodal network structure containing several groups of branch functions; Based on the output of the bimodal network structure and the text requirements, the text network infrastructure is selected, and the channel deletion operation is performed on the network infrastructure, and the long short-term memory network is spliced ​​to obtain the text semantic network structure; The bimodal network structure is integrated with the text semantic network structure to obtain the automatic recognition network structure.

[0016] Preferably, the tomographic image and the answer annotation are input into the automatic recognition network structure to perform network training, and the evaluation accuracy of the output vertebral area fracture information is verified to include: After the answer annotation text is converted into word vectors, the tomographic images are combined to divide the training set and the validation set according to a preset ratio, and the training set is input into the automatic recognition network structure; Use the learning rate adjustment strategy to set the initial learning rate to start the initial training of automatic recognition network structure, and increase the learning rate for normal training after reaching the number of traversals; During the training process of the automatic recognition network structure, the stability of the loss function is judged, and after the loss function is stable, the structure that meets the traversal times requirement is selected as the automatic recognition network structure that has completed the training; The validation set is input into the trained automatic recognition network structure, and the branching function is used to output the detection frame of the vertebral body area, the number of pixels in the vertebral sub-area segmentation, and the fracture classification information; The output information is compared with the answer annotation text, and the evaluation accuracy of the output vertebral area fracture information is verified based on the comparison results.

[0017] Preferably, comparing the output information with the answer annotation text, and verifying the evaluation accuracy of the output vertebral area fracture information based on the comparison result includes: The overlap between the detection frame and the corresponding main area frame in the answer annotation text is judged, and the percentage between the number of segmented pixels of the vertebral sub-region and the corresponding pixel-level annotation result in the answer annotation text is calculated; Compare the fracture classification information with the classification information calibration structure in the answer annotation text, and count the distances between each vertebral sub-region based on the sub-region pixels; The comparison results, overlap judgment results and percentage calculation results are integrated to analyze the output accuracy of the trained automatic recognition network structure.

[0018] Preferably, the fracture information identification and judgment unit includes: The fracture severity analysis module is used to analyze the size and location of the fracture based on the fracture information, determine the distance between each vertebral joint and the adjacent sub-joints, and analyze the severity of the vertebral fracture; The fracture impact degree analysis module is used to input the severity of vertebral fractures and the answer annotation text into the automatic recognition network structure to obtain the fracture impact degree; The mobile terminal display module is used to convert the automatic recognition network structure into a mobile terminal computing framework using quantitative compression processing technology, and then upload the severity analysis results and the degree of fracture impact to the mobile terminal.

[0019] The beneficial effects of the present invention are: The present invention proposes an artificial intelligence-based automatic identification and analysis system for vertebral fractures through artificial intelligence technology. It uses artificial intelligence algorithms to analyze the size, location and type of fractures to determine the severity, evaluate the possible impact on patients, and give follow-up treatment recommendations. The actual operation is convenient and simple, and has important promotion and use value. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0021] Figure 1 It is a principle block diagram of an automatic identification and analysis system for vertebral fractures based on artificial intelligence according to an embodiment of the present invention.

[0022] Figure 2 It is a schematic flow chart of an automatic identification and analysis system of vertebral fractures based on artificial intelligence according to an embodiment of the present invention.

[0023] Figure 3 It is a flow chart of the network structure of a dual-modal large model in an automatic identification and analysis system of vertebral fractures based on artificial intelligence according to an embodiment of the present invention.

[0024] Figure 4 It is a flowchart of the network structure of a text semantics large model in an artificial intelligence-based vertebral fracture automatic recognition and analysis system according to an embodiment of the present invention.

[0025] In the figure: 1. Scanning image processing and annotation unit; 2. Vertebral fracture automatic identification unit; 3. Fracture information identification and judgment unit. DETAILED DESCRIPTION

[0026] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.

[0027] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0028] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor is it a separate or selective embodiment that is mutually exclusive with other embodiments.

[0029] The present invention is described in detail with reference to schematic diagrams. When describing the embodiments of the present invention, for the sake of convenience, the cross-sectional diagrams showing the device structure will not be partially enlarged according to the general scale, and the schematic diagrams are only examples, which should not limit the scope of protection of the present invention. In addition, in actual production, the three-dimensional dimensions of length, width and depth should be included.

[0030] At the same time, in the description of the present invention, it should be noted that the directions or positional relationships indicated by the terms "upper, lower, inner and outer" are based on the directions or positional relationships shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore cannot be understood as limiting the present invention. In addition, the terms "first, second or third" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0031] In the present invention, unless otherwise clearly specified and limited, the terms "install, connect, connect" should be understood in a broad sense, for example: it can be a fixed connection, a detachable connection or an integral connection; it can also be a mechanical connection, an electrical connection or a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0032] Embodiment 1, as Figure 1-Figure 4 As shown, according to an embodiment of the present invention, an automatic identification and analysis system for vertebral fractures based on artificial intelligence, the automatic identification and analysis system for vertebral fractures comprises: The scanning image processing and annotation unit 1 is used to perform a control and processing operation on the tomographic scanning image, and to annotate the answer based on the tomographic scanning image after the processing.

[0033] It should be noted that the purpose of regulating and processing the patient CT images (tomographic scans) collected by the mobile terminal is to make the images sent to the deep learning network as rich as possible. In actual use, when the mobile app takes CT images, there are interferences from environmental factors such as different backgrounds, different lighting, and different angles, which makes the subsequent analysis effect unsatisfactory. Therefore, in order to better robustly capture images in different environments and achieve better analysis results, it is necessary to preprocess the CT images collected by the mobile app, including but not limited to: image rotation, image noise, image affine transformation, image scaling and other operations.

[0034] At the same time, the processed images are annotated with answers, including the vertebral main area box (rectangular box) in the CT image, mask annotation of each sub-area of ​​the vertebral body (pixel-level annotation), whether there is a fracture, the patient's weight, age, injury mechanism, and fracture history, as well as information such as the corresponding healing time, stability of the fracture site, and risk of secondary fracture based on the severity of the fracture.

[0035] In this embodiment, the scanned image processing and labeling unit 1 includes: An image rotation enhancement module is used to perform rotation processing on the tomographic image using image rotation technology and implement image enhancement operations after the rotation is completed; An image affine scaling module is used to perform affine transformation processing on the tomographic image and to scale the tomographic image using bilinear interpolation technology after the affine transformation; The scanned image annotation module is used to annotate the answers of the processed tomographic scan images according to the input form of the deep learning network.

[0036] Specifically, when the tomographic image is rotated using the image rotation technology and the image enhancement operation is performed after the rotation is completed, the tomographic image can be rotated counterclockwise according to a predefined rotation point using the image rotation technology, and the rotation is stopped after rotating to a preset angle; the initial coordinate position of the tomographic image is combined with the predefined rotation point to obtain the coordinate position of the rotated tomographic image; the image quality of the rotated tomographic image is evaluated, and the evaluation result is merged with the noise generation technology to add noise to the tomographic image; the tomographic image to which the noise is added is inversely transformed to complete the noise removal operation, thereby obtaining the enhanced tomographic image.

[0037] It should be explained that image rotation refers to rotating a defined tomographic image around a certain point in a counterclockwise or clockwise direction by a certain angle, usually refers to rotating around the center of the tomographic image in a counterclockwise direction.

[0038] Assuming that the upper left corner of the tomographic image is (left, top) and the lower right corner is (right, bottom), then any point (x 0 ,y 0 ) around its center (x center ,y center ) After rotating counterclockwise by angle angle, the calculation formula for the new coordinate position (x′, y′) is: ; ; in, ; ; In the formula, θ represents the counterclockwise rotation angle of the tomographic image, and in order to better robust the scene in actual use, the rotation angle of the tomographic image is controlled within plus or minus 15 degrees.

[0039] In addition, since various factors in the actual shooting process often cause various noise points to be stored in the captured tomographic images, in order to be robust to different noise environments, it is necessary to add and remove image noise from the captured tomographic images.

[0040] In the process of generating noise for an image, a variety of random noise combinations (including but not limited to Gaussian noise, salt and pepper noise, Gaussian filtering, mean filtering, gamma noise, etc.) are performed simultaneously, and image noise removal is to perform an inverse transformation on the image with noise added to obtain an image after noise removal.

[0041] Specifically, when the tomographic image is subjected to affine transformation and the tomographic image is scaled using bilinear interpolation technology after the affine transformation, the type and parameters of the affine transformation can be determined based on the tomographic image, and then a transformation matrix can be constructed, and the mapping position of the tomographic image can be defined using the transformation matrix; the tomographic image is subjected to affine transformation using the transformation matrix relationship to obtain a mapping coordinate matrix of the changed tomographic image; and the tomographic image is enlarged or scaled in the longitudinal and transverse directions based on the bilinear interpolation technology and the tomographic image.

[0042] It needs to be explained that affine transformation, also known as affine mapping, refers to a linear transformation of a vector space in geometry followed by a translation to transform it into another vector space. Affine transformation can maintain the "straightness" of the image, including rotation, scaling, translation, and shearing operations. The common affine transformation matrix is ​​a 2*3 matrix with two rows and three columns. The elements in the third column play a role of translation, the numbers on the diagonal of the first two columns are scaling, and the rest are rotation or shearing. The transformation matrix relationship is as follows: ; Among them, A represents the transformed coordinate matrix, B represents the affine transformation matrix, and C represents the original coordinate matrix. There are 6 unknown quantities. Assuming that the tomographic image is rotated clockwise by σ radians with (m, n) as the axis to the target image, the variables corresponding to the transformation matrix are: ; Among them, the four unknown variables a, b, d, and e in the first two columns play a rotation role, and the two unknown variables c and f in the third column play a translation role.

[0043] At the same time, in actual use, due to various differences in the distance of the image, the actual size of the vertebra in the tomographic image is not fixed. The bilinear interpolation method is used to simulate the sizes of different vertebral regions. That is, through the bilinear interpolation method, each tomographic image is randomly transformed, enlarged or reduced, so that the training data is richer. The specific implementation of the bilinear method is as follows: Assume that the value of function g at point P = (i, j) and it is known that function g at Q 11 =(i 1 , j 1 )、Q 12 =(i 1 , j 2 )、Q 21 =(i 2 , j 1 )、Q 22 =(i 2 , j 2 ) The values ​​of the four points, then R 1 and R 2 are the interpolation values ​​in the i (horizontal) direction respectively.

[0044] First, perform linear interpolation in the i direction and obtain: ; ; ; ; Then perform linear interpolation in the j direction (longitudinal direction) to obtain:

[0045] Specifically, when the processed tomographic image is annotated with answers according to the input form of the deep learning network, the main area frame of the vertebra in the tomographic image can be annotated based on the rectangular annotation technology to obtain the complete area of ​​the vertebra, and the pixel-level annotation is used to perform secondary annotation in the complete area of ​​the vertebra to obtain the vertebral sub-area; the vertebral sub-areas are digitally labeled in the labeling order, and after the labeling is completed, the vertebral sub-areas are calibrated with classified information to determine the fracture status of the vertebra; the personal information, injury mechanism and fracture history of the person corresponding to the tomographic image are obtained according to the image acquisition information, and the fracture status assessment annotation is performed on the corresponding tomographic image in combination with the expert annotation technology; the personal information, injury mechanism, fracture history and fracture status assessment annotation results of the corresponding person are integrated into the answer annotation text.

[0046] It needs to be explained that according to the data format required by the deep learning network, the tomographic image needs to be annotated. Specifically, the vertebral body area box in the CT image needs to be annotated, that is, the circumscribed rectangle of the vertebral body in the image, where the circumscribed rectangle needs to completely include the vertebral body area. At the same time, in order to better analyze the fracture information, the tomographic image also needs to be masked (pixel-level labeled) for each vertebral sub-area, that is, each vertebral part needs to be labeled, for example, the area where the vertebral body 1 is located is labeled as 1, the area where the vertebral body 2 is located is labeled as 2, and so on for labeling other sub-areas.

[0047] At the same time, it is necessary to calibrate the classification information of all tomographic images, that is, to determine whether there is a fracture in the current tomographic image. If so, it is calibrated as mild fracture, moderate fracture, or severe fracture. If not, it is calibrated as none, and other cases are marked as unknown. Based on the calibration of the current CT image corresponding to the patient's weight, age, injury mechanism and fracture history, the specific injury mechanism mainly includes falls, fights, collisions, etc. The fracture history mainly includes whether there is a fracture within 3 months, whether there is a fracture within 3 to 6 months, whether there is a fracture within 6 to 9 months, and whether there is a fracture within 9 to 12 months.

[0048] The corresponding healing time, stability of the fracture site, and risk of secondary fracture are calibrated according to the severity of the fracture. Specifically, each tomography image can be calibrated by a chief physician with rich clinical experience. Based on personal experience, a preliminary evaluation answer of the current CT image can be given, including healing time (1 to 3 months, 3 to 6 months, 6 to 9 months, 9 to 12 months), stability of the fracture site (extremely unstable, stable, slightly stable), and risk of secondary fracture (high, medium, low).

[0049] The vertebral fracture automatic identification unit 2 is used to construct an automatic identification network structure, train the automatic identification network structure using tomographic images and answer annotations, and identify fracture information using the trained automatic identification network structure.

[0050] In this embodiment, the vertebral fracture automatic identification unit 2 includes: The network structure building module is used to build a bimodal network and a large text semantic model structure, and automatically identify the network structure based on the construction results; Structural parameter configuration module, used to configure the hyper parameters of automatic identification network structure and training files; The dual-modal network training module is used to input the tomographic images and answer annotations into the automatic recognition network structure to perform network training and verify the evaluation accuracy of the output vertebral area fracture information; The network structure adjustment module is used to adjust the automatic recognition network structure based on the evaluation accuracy results.

[0051] Specifically, when constructing a bimodal network and a large text semantic model structure and obtaining an automatic recognition network structure based on the construction result, a deep convolutional neural network model can be selected as the basic network structure for feature extraction, and a scale feature layer is set in combination with a pyramid structure, and convolution and deconvolution operations are performed on each scale feature layer; after the convolution is completed, a feature fusion operation is performed, and the branch function of each scale feature layer is defined to obtain a bimodal network structure containing several groups of branch functions; based on the output result of the bimodal network structure and the text requirements, a text network infrastructure is selected, and a channel deletion operation is performed on the network infrastructure, and then a long short-term memory network is spliced ​​to obtain a text semantic network structure; the bimodal network structure is fused with the text semantic network structure to obtain an automatic recognition network structure.

[0052] It should be explained that the bimodal network structure adopts a multimodal structure in which text and images are input simultaneously. The network output results also include detection boxes, segmentation mask maps, classification results, etc. It is a multi-task network structure. Specifically, the network structure includes three branches: detection branch, segmentation branch, and classification branch.

[0053] In order to be suitable for mobile terminal operation, an improved version of ResNet18 is used as the backbone (basic network structure), and a multi-layer pyramid structure is adopted, which includes multiple scales such as 1 / 2, 1 / 4, 1 / 8, 1 / 16, and 1 / 32. The feature layers of the 1 / 8, 1 / 16, and 1 / 32 scales are subjected to multiple convolution and deconvolution operations and then feature fusion is performed as the detection branch; the feature layers of the 1 / 2, 1 / 4, and 1 / 8 scales are subjected to multiple convolution and deconvolution operations and then feature fusion is performed as the segmentation branch; after the 1 / 32 scale, multiple convolution layers, pooling layers, regularization layers, etc. are spliced ​​as the classification branch.

[0054] Text information is input into the network structure, and big data modeling is used to output the severity of fractures, healing time, stability of fracture sites, and risk of secondary fractures in each tomographic image.

[0055] First, the network structure was modified based on the original inception-v3, including deleting some channels and some convolutional layers to improve the network operation speed. Secondly, the bidirectional LSTM structure was spliced. During the modeling process, the summarized text information was converted into word vectors and sent to the constructed network structure for training. The specific text information included fracture location, fracture severity, treatment recommendations, patient age, weight, fracture history, fracture mechanism, healing time, stability of the fracture site, risk of secondary fracture, and other information.

[0056] Specifically, the tomographic image and the answer annotation are input into the automatic recognition network structure to perform network training. When verifying the evaluation accuracy of the output vertebral area fracture information, the answer annotation text can be converted into a word vector and combined with the tomographic image to divide the training set and the verification set into a preset ratio, and the training set is input into the automatic recognition network structure; the learning rate adjustment strategy is used to set the initial learning rate to start the preliminary training of the automatic recognition network structure, and the learning rate is increased after the number of traversals is reached to perform normal training; during the training of the automatic recognition network structure, the stability of the loss function is judged, and after the loss function is stable, the structure that meets the traversal number requirement is selected as the trained automatic recognition network structure; the verification set is input into the trained automatic recognition network structure, and the branch function is used to output the detection frame of the vertebral main area, the number of pixels of the vertebral sub-area segmentation and the fracture classification information; the output information is compared with the answer annotation text, and the evaluation accuracy of the output vertebral area fracture information is verified based on the comparison result.

[0057] It should be explained that the super parameters in the network structure are set according to the initial framework of the network structure, including the size of the input tomographic image, the number of channels of each convolutional layer in the backbone, the size of the convolution kernel, and the setting of convolutional layers between multiple scales such as 1 / 2, 1 / 4, 1 / 8, 1 / 16, and 1 / 32.

[0058] Among them, the size of the input tomographic image is set to 960*960, the number of channels of the first four convolutional layers in the backbone is 64, and the feature layer size is 960*960; the number of channels of the fifth to eighth convolutional layers is 128, and the feature layer size is 480*480; the number of channels of the ninth to twelfth convolutional layers is 256, and the feature layer size is 240*240; the number of channels of the thirteenth to sixteenth convolutional layers is 256, and the feature layer size is 120*120; the number of channels of the seventeenth to twentieth convolutional layers is 512, and the feature layer size is 60*60; the convolution kernel size of all convolutional layers is 3*3, and the step size is 2 in each scale transformation layer to achieve downsampling, and each convolutional layer is connected to the BN layer, Scale layer and Relu layer. Where multiple scales are fused, the Concat layer is used to merge channels.

[0059] In order to make the network model converge better and obtain a stable model, it is necessary to configure the super parameters of the answer annotation text during the training process. The optimizer uses SGD (stochastic gradient descent), the learning rate uses CosineAnnealingLR (cosine annealing adjusted learning rate), Tmax (maximum number of iterations) is 10, the basic learning rate is 0.0001, the number of epochs (number of traversals) is 300, and 8 T4 graphics cards are used for parallel training. The number of batch processing (batch processing) of each graphics card is 64. The loss functions are BCE loss, CIOU loss, CrossEntropy loss, etc.

[0060] The calibrated text information is converted into word vectors and sent to the established network structure together with the image for training. The text information here includes fracture conditions, fracture areas, various sub-module areas of the spine, the front and back heights of each area, and the body's own height for network model training.

[0061] At the beginning of training, the warmup strategy is first adopted (the model is relatively unstable at the beginning of training, and the initial learning rate should be set very low to ensure that the network has good convergence, but a low learning rate will make the training process very slow, so the method of gradually increasing the learning rate from a low learning rate to a higher learning rate is used to achieve the "warm-up" stage of network training, which is called warmup). Use a small learning rate for preliminary training, and perform normal training after 20 epochs.

[0062] Specifically, the output information is compared with the answer annotation text, and the evaluation accuracy of the output vertebral area fracture information is verified based on the comparison result. The overlap between the detection box and the corresponding main area box in the answer annotation text can be judged, and the percentage between the number of vertebral sub-area segmentation pixels and the corresponding pixel-level annotation result in the answer annotation text is calculated; the fracture classification information is compared with the classification information calibration structure in the answer annotation text, and the distance between each vertebral sub-area is counted based on the body sub-area pixels; the comparison result, the overlap judgment result and the percentage calculation result are integrated to analyze the output accuracy of the trained automatic recognition network structure.

[0063] When the loss is stable, take the model after 300 epochs and verify the effect on the test set (the data in the test set is not involved in the model training), including the overlap between the vertebral area frame and the calibrated answer (that is, the overlap between the detection frame of the vertebral area output by the model and the frame of the area manually calibrated in the image, that is, the IOU between the detection frame predicted by the model and the manually calibrated answer frame. The higher the overlap, the better the model effect, and the lower the overlap, the worse the model effect). The overlap thresholds are 0.3, 0.4, 0.5, and 0.6 respectively. The detection frame accuracy under different overlap thresholds is tested. The effect of the segmentation branch is checked, mainly by calculating the image effectively segmented in each sub-network structure. The percentage between the number of prime values ​​and the calibrated answer (for example, the current area has N1 pixel values, and these N1 pixels are the manually calibrated answers. The values ​​predicted by the network structure are compared with these 100. If the network structure predicts N2 values, the percentage is N2 / N1). Whether there is a classification branch for fractures, the network structure outputs two-dimensional information in this branch: whether there is a fracture or not. The result is directly compared with the calibrated answer. If the result output by the network structure is consistent with the calibrated answer, the image analysis is considered correct. If not, the network structure analysis is considered wrong. The final accuracy is calculated. In addition, the distance between each sub-area is calculated, compared with the accuracy of the calibrated answer, and the percentage is calculated.

[0064] The summarized text information is converted into word vectors and sent to the established network structure for training. The text here includes: information including fracture location, fracture severity, treatment recommendations, patient age, weight, fracture history, fracture mechanism, healing time, stability of the fracture site, risk of secondary fracture, etc. After network structure training, the network structure can output the severity of the fracture, healing time, risk of secondary fracture and related treatment recommendations in the current CT image.

[0065] The fracture information identification and judgment unit 3 is used to analyze the severity of the fracture according to the fracture information, evaluate the impact of the fracture using the severity analysis result, and transmit the severity analysis result and the impact of the fracture to the mobile terminal.

[0066] In this embodiment, the fracture information identification and judgment unit 3 includes: The fracture severity analysis module is used to analyze the size and location of the fracture based on the fracture information, determine the distance between each vertebral joint and the adjacent sub-joints, and analyze the severity of the vertebral fracture; The fracture impact degree analysis module is used to input the severity of vertebral fractures and the answer annotation text into the automatic recognition network structure to obtain the fracture impact degree; The mobile terminal display module is used to convert the automatic recognition network structure into a mobile terminal computing framework using quantitative compression processing technology, and then upload the severity analysis results and the degree of fracture impact to the mobile terminal.

[0067] Specifically, in order to facilitate CT image recognition, the network structure is applied to the mobile terminal. In order to enable the network structure to run better on the mobile terminal, the network structure has been lightweight when it is constructed, such as the number of channels, the number of network layers, etc. During use, the trained network model is converted into an ncnn int8 model to improve the running speed while ensuring the reasoning accuracy. Based on the Java language, a mobile-based component code is developed, which can capture images and display effects on the mobile terminal, including whether there is a fracture, height loss, and treatment recommendations.

[0068] It should be explained that the corresponding degree of fracture can be obtained by processing the relevant output information based on the network structure, including the main vertebral area, the mask map of each sub-area and the corresponding attribute classification results.

[0069] The details are as follows: The vertebral body region frame can be used to obtain the vertebral body region image in the CT image. During the training process, this region frame not only helps the network to better focus on the vertebral body region and prevent interference from other regions, but also helps to regress the masks of each sub-joint. Secondly, the masks of each vertebral body sub-region are obtained on the acquired vertebral body region image and the mask output by the network model, such as the complete region of the first vertebra and the complete region of the second vertebra. The complete region here refers to the pixel-level region of each vertebra.

[0070] Based on the above results, all the segmentation results of each sub-joint in the entire vertebra are obtained, that is, the segmentation map of each sub-region of the vertebra. The size of each sub-joint or vertebra of the vertebra or the distance between adjacent vertebrae of a normal person remains unchanged. According to the mask map of each sub-joint in the network output, the distance between each sub-joint or vertebra and its adjacent sub-joint is calculated, such as the distance between the second section and the first section, and the distance between the second section and the third section. The distance is the height.

[0071] By analogy, by calculating the anterior and posterior heights between other sub-joints, we can get the anterior, middle, and posterior heights of each sub-joint in the entire vertebra. The anterior height here refers to the distance between the current sub-joint and the previous sub-joint, the middle height refers to the height of each sub-joint, and the posterior height refers to the distance between the current sub-joint and the next sub-joint, and the degree of fracture of the current vertebra can be obtained. At the same time, the attribute classification information output by the network structure is further weighted to confirm the degree of vertebral fracture. The classification items include: unknown, mild fracture, moderate fracture, and severe fracture. The attribute classification results and the judgment results are weighted to output the final vertebral fracture situation. The weighting here means that the attribute classification result score exceeds 0.95, which means that the result is considered correct. At the same time, combined with the judgment results of each sub-joint mask, the final result is given, and the corresponding treatment recommendations can be given through the analysis of the degree of vertebral fracture.

[0072] Specifically, the degree of vertebral fracture is divided into mild fracture, moderate fracture and severe fracture, and the criteria for judgment are as follows: Mild fracture: Compared with adjacent or same vertebrae, the height of the anterior, middle and posterior parts of the vertebrae decreases by less than 25%; moderate fracture: Compared with adjacent or same vertebrae, the height of the anterior, middle and posterior parts of the vertebrae decreases by 25%-40%; severe fracture: Compared with adjacent or same vertebrae, the height of the anterior, middle and posterior parts of the vertebrae decreases by more than 40%.

[0073] Based on the fracture assessment and treatment recommendations, combined with the patient's current relevant information: age, weight, fracture history, etc., all information is sent into the network structure to obtain more comprehensive information.

[0074] When it is a mild fracture, the healing period is 1-3 months, the risk of secondary fracture is low, and the recommended treatment is: bed rest and conservative treatment.

[0075] When it is a moderate fracture, the healing period is 3-6 months, the risk of secondary fracture is moderate, and the recommended treatment is: vertebroplasty and minimally invasive surgery.

[0076] When it is a severe fracture, the healing period is 3-6 months, and the risk of secondary fracture is high. The recommended treatment is: vertebral correction and open surgery.

[0077] To sum up, with the help of the above-mentioned technical scheme of the present invention, the present invention proposes an artificial intelligence-based vertebral fracture automatic identification and analysis system through artificial intelligence technology. It uses artificial intelligence algorithms to analyze the size, location and type of fractures to judge the severity, evaluate the possible impact on patients, and give follow-up treatment recommendations. The actual operation is convenient and concise, and it has important promotion and use value.

[0078] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. The automatic identification and analysis system of vertebral fracture based on artificial intelligence is characterized by: include, A scanning image processing and annotation unit, used for performing a control and processing operation on the tomographic scanning image, and annotating the answer based on the tomographic scanning image after the processing; Vertebral fracture automatic recognition unit, used to construct an automatic recognition network structure, train the automatic recognition network structure using tomographic images and answer annotations, and recognize fracture information using the trained automatic recognition network structure; The fracture information recognition and judgment unit is used to analyze the severity of the fracture based on the fracture information, use the severity analysis results to evaluate the impact of the fracture, and transmit the severity analysis results and the impact of the fracture to the mobile terminal.

2. The artificial intelligence-based vertebral fracture automatic identification and analysis system according to claim 1, characterized in that: The scanned image processing and annotation unit comprises: An image rotation enhancement module is used to perform rotation processing on the tomographic image using image rotation technology and implement image enhancement operations after the rotation is completed; An image affine scaling module is used to perform affine transformation processing on the tomographic image and to scale the tomographic image using bilinear interpolation technology after the affine transformation; The scanned image annotation module is used to annotate the answers of the processed tomographic scan images according to the input form of the deep learning network.

3. The artificial intelligence-based vertebral fracture automatic identification and analysis system according to claim 2, characterized in that: The method of performing rotation processing on the tomographic image by using the image rotation technology and performing image enhancement operation after the rotation is completed includes: The tomographic image is rotated counterclockwise according to a predefined rotation point using an image rotation technology, and the rotation stops after reaching a preset angle; Combining the initial coordinate position of the tomographic image with the predefined rotation point to obtain the coordinate position of the rotated tomographic image; evaluating the image quality of the rotated tomographic image and fusing the evaluation result with a noise generation technique to add noise to the tomographic image; The noise-added tomographic image is subjected to inverse transformation to complete the noise removal operation, thereby obtaining an enhanced tomographic image.

4. The artificial intelligence-based vertebral fracture automatic identification and analysis system according to claim 3, characterized in that: The affine transformation of the tomographic image and the scaling of the tomographic image using bilinear interpolation technology after the affine transformation include: After determining the type and parameters of the affine transformation based on the tomographic image, a transformation matrix is ​​constructed, and a mapping position of the tomographic image is defined by using the transformation matrix; Performing affine transformation on the tomographic image using the transformation matrix relationship to obtain a mapping coordinate matrix of the changed tomographic image; Based on the bilinear interpolation technology and the tomographic image, the tomographic image is enlarged or scaled in the longitudinal and transverse directions.

5. The artificial intelligence-based vertebral fracture automatic identification and analysis system according to claim 4, characterized in that: The answer labeling process for the processed tomographic image according to the input form of the deep learning network includes: Based on the rectangular annotation technology, the main area frame of the vertebra in the tomographic image is annotated to obtain the complete area of ​​the vertebra, and the pixel-level annotation is used to perform secondary annotation in the complete area of ​​the vertebra to obtain the vertebral sub-area; The vertebral sub-regions are digitally labeled in the order of labeling, and after the labeling is completed, the classification information of the vertebral sub-regions is calibrated to determine the fracture status of the vertebral body; Obtain personal information, injury mechanism and fracture history of the person corresponding to the tomographic image based on the image acquisition information, and combine the corresponding tomographic image with expert annotation technology to perform fracture status assessment and annotation; The personal information, injury mechanism, fracture history and fracture status assessment annotation results of the corresponding person are integrated into the answer annotation text.

6. The artificial intelligence-based automatic identification and analysis system for vertebral fractures according to claim 5, characterized in that: The vertebral fracture automatic identification unit comprises: The network structure building module is used to build a bimodal network and a large text semantic model structure, and automatically identify the network structure based on the construction results; Structural parameter configuration module, used to configure the hyper parameters of automatic identification network structure and training files; The recognition network structure training module is used to input the tomography image and the answer annotation into the automatic recognition network structure to perform network training and verify the evaluation accuracy of the output vertebral area fracture information; The network structure adjustment module is used to adjust the automatic recognition network structure based on the evaluation accuracy results.

7. The artificial intelligence-based automatic identification and analysis system for vertebral fractures according to claim 6, characterized in that: The step of constructing a bimodal network and a text semantics large model structure and obtaining an automatic recognition network structure based on the construction result includes: The deep convolutional neural network model is selected as the basic network structure for feature extraction, the scale feature layer is set in combination with the pyramid structure, and convolution and deconvolution operations are performed on each scale feature layer; After the convolution is completed, a feature fusion operation is performed, and the branch functions of the feature layers at each scale are defined to obtain a bimodal network structure containing several groups of branch functions; Based on the output of the bimodal network structure and the text requirements, the text network infrastructure is selected, and the channel deletion operation is performed on the network infrastructure, and the long short-term memory network is spliced ​​to obtain the text semantic network structure; The bimodal network structure is integrated with the text semantic network structure to obtain the automatic recognition network structure.

8. The artificial intelligence-based vertebral fracture automatic identification and analysis system according to claim 7, characterized in that: The tomographic image and the answer annotation are input into the automatic recognition network structure to perform network training, and the evaluation accuracy of the output vertebral area fracture information is verified, including: After the answer annotation text is converted into word vectors, the tomographic images are combined to divide the training set and the validation set according to a preset ratio, and the training set is input into the automatic recognition network structure; Use the learning rate adjustment strategy to set the initial learning rate to start the initial training of automatic recognition network structure, and increase the learning rate for normal training after reaching the number of traversals; During the training process of the automatic recognition network structure, the stability of the loss function is judged, and after the loss function is stable, the structure that meets the traversal times requirement is selected as the automatic recognition network structure that has completed the training; The validation set is input into the trained automatic recognition network structure, and the branching function is used to output the detection frame of the vertebral body area, the number of pixels in the vertebral sub-area segmentation, and the fracture classification information; The output information is compared with the answer annotation text, and the evaluation accuracy of the output vertebral area fracture information is verified based on the comparison results.

9. The artificial intelligence-based vertebral fracture automatic identification and analysis system according to claim 8, characterized in that: The output information is compared with the answer annotation text, and the evaluation accuracy of the output vertebral area fracture information is verified based on the comparison result, including: The overlap between the detection frame and the corresponding main area frame in the answer annotation text is judged, and the percentage between the number of segmented pixels of the vertebral sub-region and the corresponding pixel-level annotation result in the answer annotation text is calculated; Compare the fracture classification information with the classification information calibration structure in the answer annotation text, and count the distances between each vertebral sub-region based on the sub-region pixels; The comparison results, overlap judgment results and percentage calculation results are integrated to analyze the output accuracy of the trained automatic recognition network structure.

10. The artificial intelligence-based automatic identification and analysis system for vertebral fractures according to claim 9, characterized in that: The fracture information identification and judgment unit comprises: The fracture severity analysis module is used to analyze the size and location of the fracture based on the fracture information, determine the distance between each vertebral joint and the adjacent sub-joints, and analyze the severity of the vertebral fracture; The fracture impact degree analysis module is used to input the severity of vertebral fractures and the answer annotation text into the automatic recognition network structure to obtain the fracture impact degree; The mobile terminal display module is used to convert the automatic recognition network structure into a mobile terminal computing framework using quantitative compression processing technology, and then upload the severity analysis results and the degree of fracture impact to the mobile terminal.

Images