Bone lesion classification method and device based on whole-body bone scanning image and clinical information, equipment and medium

Abstract

The invention discloses a bone lesion classification method and device based on whole-body bone scanning images and clinical information, equipment and a medium. The method comprises the steps that the whole-body bone scanning images and the clinical information of at least two visual angles are acquired; extracting whole-body bone scanning features of the whole-body bone scanning image at each view angle through a feature extraction module, and extracting clinical feature vectors of clinical information through a clinical information embedding module; fusing all the whole body bone scanning features through a feature fusion module to obtain a fused feature vector; and determining a bone lesion category based on the fusion feature vector and the clinical feature vector through a classification head. According to the method, at least two visual angles are adopted to scan the images of the whole body bone, lesion missing detection caused by tissue overlapping in a single visual angle can be avoided, anatomical structures of front and rear visual angles have natural symmetry, spatial position correspondence of the same lesion in different visual angles can be ensured, accurate spatial reference is provided for feature fusion, and the accuracy of feature fusion is improved. And the prediction accuracy of bone lesion categories is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of biomedical technology, and in particular to a bone lesion classification method and device based on whole-body bone scan images and clinical information, equipment and medium. BACKGROUND

[0002] Bone scintigraphy, also known as bone scan, is a common tumor examination method in nuclear medicine departments, which has high sensitivity for bone lesions. Phosphate substances such as methylene diphosphonate (MDP) have the characteristics of high uptake in the skeleton and do not remain in soft tissues, so they are used as radioactive tracers for bone scanning after being labeled with 99mTc. By detecting the changes in tracer uptake, whole-body bone scintigraphy can provide important information about the location, prognosis and treatment effect of bone lesions.

[0003] In clinical practice, bone scintigraphy is often used to diagnose suspected bone diseases such as bone metastases, primary bone tumors, bone and joint inflammation, bone trauma, etc. Bone scanning has the advantage of high sensitivity, which can detect bone lesions earlier, but also has the limitation of low specificity, which makes it difficult to distinguish between benign lesions and bone metastases. For example, multiple benign osteolytic lesions (eosinophilic granuloma, fibrous dysplasia and enchondroma) have similar manifestations in bone imaging as bone metastases, which can easily lead to misdiagnosis and false positive diagnosis; local accumulation of radioactive tracers caused by spinal degenerative diseases is also difficult to distinguish from bone metastases. In addition, the review of bone scan images requires the rich experience of imaging physicians, and there may be consistency problems between different physicians.

[0004] Therefore, the prior art still needs to be improved and improved. SUMMARY

[0005] The technical problem to be solved by the present application is to provide a bone lesion classification method, device, equipment and medium based on whole-body bone scan images and clinical information, which addresses the shortcomings of the prior art.

[0006] To solve the above technical problems, the first aspect of the present application provides a bone lesion classification method based on whole-body bone scan images and clinical information, which applies a trained bone lesion detection model, the bone lesion detection model includes a feature extraction module, a feature fusion module, a classification head and a clinical information embedding module; the method comprises:

[0007] Obtaining at least two perspective whole-body bone scan images and clinical information;

[0008] Extracting whole-body bone scan features of each perspective whole-body bone scan image through the feature extraction module, and extracting a clinical feature vector of the clinical information through the clinical information embedding module;

[0009] fuse each whole body bone scan feature to obtain a fusion feature vector through the feature fusion module;

[0010] determine a bone lesion category based on the fusion feature vector and the clinical feature vector through the classification head.

[0011] The bone lesion classification method based on whole body bone scan images and clinical information, wherein the feature extraction module comprises N cascaded feature extraction units; and the extracting whole body bone scan features of each view whole body bone scan image through the feature extraction module specifically comprises:

[0012] For each view whole body bone scan image, input the view whole body bone scan image into a first feature extraction unit, and extract a first feature map through the first feature extraction unit;

[0013] input the first feature map into a second feature extraction unit, extract a second feature map through the second feature extraction unit, and so on until an N-1th feature map is extracted through an N-1th feature extraction unit;

[0014] splice the N-1th feature map to an N-Mth feature map, input the spliced feature map into an Nth feature extraction unit, and extract the whole body bone scan features of the view whole body bone scan image through the Nth feature extraction unit.

[0015] The bone lesion classification method based on whole body bone scan images and clinical information, wherein the feature fusion module comprises a scaled dot-product attention unit and a fusion unit, and the fusing each whole body bone scan feature to obtain a fusion feature vector through the feature fusion module specifically comprises:

[0016] generate a query vector, a key vector and a value vector based on each whole body bone scan feature;

[0017] generate a query vector, a key vector and a value vector based on each whole body bone scan feature, determine a self-attention feature vector through the scaled dot-product attention unit based on the query vector generated based on each whole body bone scan feature and the key vector and the value vector generated based on each other whole body bone scan feature, and determine a cross-attention feature vector through the scaled dot-product attention unit based on the query vector generated based on each whole body bone scan feature and the key vector and the value vector generated based on each other whole body bone scan feature;

[0018] fuse all self-attention feature vectors and all cross-attention feature vectors through the fusion unit to obtain a fusion feature vector.

[0019] The bone lesion classification method based on whole-body bone scan images and clinical information, wherein the scaled dot-product attention unit comprises a first matrix multiplication layer, a scaling layer, a normalization layer, and a second matrix multiplication layer; the first matrix multiplication layer, the scaling layer, the normalization layer, and the second matrix multiplication layer are sequentially cascaded, the input of the first matrix multiplication layer is a query vector and a key vector, and the input of the second matrix multiplication layer is the output of the normalization layer and a value vector.

[0020] The bone lesion classification method based on whole-body bone scan images and clinical information, wherein the at least two perspective whole-body bone scan images comprise an anterior perspective whole-body bone scan image and a posterior perspective whole-body bone scan image.

[0021] The bone lesion classification method based on whole-body bone scan images and clinical information, wherein the method further comprises:

[0022] obtaining a gradient map of bone lesion categories relative to each whole-body bone scan feature;

[0023] determining an influence weight of each whole-body bone scan feature according to the gradient map, and linearly combining each whole-body bone scan feature based on the influence weight to obtain a combined feature map;

[0024] extracting a positive contribution area in the combined feature map, and superimposing the positive contribution area on a perspective whole-body bone scan image to obtain a visualized contribution map.

[0025] The second aspect of the present application provides a bone lesion classification device based on whole-body bone scan images and clinical information, which applies a trained bone lesion detection model, wherein the bone lesion detection model comprises a feature extraction module, a feature fusion module, a classification head, and a clinical information embedding module; the device comprises:

[0026] an acquisition module configured to acquire at least two perspective whole-body bone scan images and clinical information;

[0027] a control module configured to extract whole-body bone scan features of each perspective whole-body bone scan image through the feature extraction module, extract a clinical feature vector of the clinical information through the clinical information embedding module, fuse each whole-body bone scan feature through the feature fusion module to obtain a fused feature vector, and determine a bone lesion category based on the fused feature vector and the clinical feature vector through the classification head.

[0028] The third aspect of the present application provides a computer readable storage medium, which stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the bone lesion classification method based on whole-body bone scan images and clinical information as described in any of the above.

[0029] The fourth aspect of the present application provides a terminal device, comprising a processor and a memory;

[0030] The memory stores a computer readable program which can be executed by the processor;

[0031] The processor implements the steps in the bone lesion classification method based on whole body bone scan images and clinical information as described above when executing the computer readable program.

[0032] Beneficial effects: Compared with the prior art, the present application provides a bone lesion classification method, device, equipment and medium based on whole body bone scan images and clinical information, the method comprising obtaining at least two perspective whole body bone scan images and clinical information; extracting whole body bone scan features of each perspective whole body bone scan image through the feature extraction module, and extracting a clinical feature vector of the clinical information through the clinical information embedding module; fusing each whole body bone scan feature to obtain a fused feature vector through the feature fusion module; and determining a bone lesion category based on the fused feature vector and the clinical feature vector through the classification head. By using at least two perspective whole body bone scan images, the present application can avoid missed detection of lesions caused by tissue overlap in a single perspective, and the anatomical structures in front and back perspectives have natural symmetry, which can ensure the spatial position correspondence of the same lesion in different perspectives, provide accurate spatial reference for feature fusion, and improve the prediction accuracy of the bone lesion category. At the same time, the feature extraction module extracts hierarchical feature expression of local details-region association-global semantics, effectively alleviates the gradient vanishing problem of deep networks, avoids loss of features of small lesions caused by continuous down-sampling, improves the recognition ability of the model for lesions of different sizes, and further improves the prediction accuracy of the bone lesion category. BRIEF DESCRIPTION OF DRAWINGS

[0033] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0034] Figure 1 The flowchart of the bone lesion classification method based on whole body bone scan images and clinical information provided by the embodiments of the present application.

[0035] Figure 2 The network structure diagram of the bone lesion detection model.

[0036] Figure 3 The principle block diagram of the bone lesion classification device based on whole body bone scan images and clinical information provided by the embodiments of the present application.

[0037] Figure 4 A principle block diagram of a terminal device provided by an embodiment of the present application is shown. DETAILED DESCRIPTION

[0038] The embodiment of the present application provides a bone lesion classification method and device based on whole-body bone scan images and clinical information, equipment and medium, so that the purpose, technical scheme and effect of the present application are more clear and definite. The present application is further described in detail below with reference to the drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and not to limit the present application.

[0039] Those skilled in the art can understand that, unless specifically stated otherwise, the singular form "a", "an", "said" and "the" used herein can also include the plural form. It should be further understood that the phrase "comprising" used in the specification of the present application means that the features, integers, steps, operations, elements and / or components exist, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we say that an element is "connected" or "coupled" to another element, it can be directly connected or coupled to the other element, or there can be intermediate elements. In addition, "connected" or "coupled" used herein can include wireless connection or wireless coupling. The phrase "and / or" used herein includes all or any single unit and all combinations of the associated listed items.

[0040] Those skilled in the art can understand that, unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meaning as that generally understood by those skilled in the art to which the present application belongs. It should also be understood that terms such as those defined in a general dictionary should be understood to have meanings consistent with those in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as such.

[0041] It should be understood that the sequence numbers and sizes of the steps in the embodiments do not mean the order of execution, and the execution order of the processes is determined by its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0042] The content of the application will be further described by describing the embodiments in combination with the drawings.

[0043] The embodiment provides a bone lesion classification method based on whole body bone scan images and clinical information, which applies a trained bone lesion detection model. The bone lesion detection model comprises a feature extraction module (FEM), a feature fusion module (BCAM), a classification head, and a clinical information embedding module (CIEM). The feature extraction module is connected with the feature fusion module. The feature fusion module and the clinical information embedding module CIEM are connected with the classification head through an adder. The feature extraction module is used to extract whole body bone scan features of the whole body bone scan images. The clinical information embedding module is used to extract a clinical feature vector of the clinical information. The feature fusion module is used to fuse each whole body bone scan feature to obtain a fusion feature vector. The classification head is used to determine a bone lesion category based on the fusion feature vector and the clinical feature vector. The embodiment of the application can avoid the inconsistency of bone lesion categories caused by different experiences of different doctors based on the whole body bone scan images and the clinical information for bone lesion category prediction. Meanwhile, the embodiment of the application extracts hierarchical feature expression of local details-region association-global semantics through the feature extraction module, effectively alleviates the gradient disappearance problem of the deep network, avoids the loss of micro-lesion features caused by continuous down-sampling, improves the recognition ability of the model for different size lesions, and further improves the prediction accuracy of the bone lesion category.

[0044] As shown in Figure 1 and 2 The bone lesion classification method based on whole body bone scan images and clinical information provided by the embodiment of the application specifically comprises:

[0045] S10, acquiring at least two perspective whole body bone scan images and clinical information.

[0046] Specifically, the at least two perspective whole body bone scan images are obtained by scanning the same patient from different perspectives, and the time interval of the scan time of the at least two perspective whole body bone scan images is less than a preset time interval. The at least two perspective whole body bone scan images can include an anterior perspective whole body bone scan image, a posterior perspective whole body bone scan image, a lateral perspective whole body bone scan image, an oblique perspective whole body bone scan image, etc.

[0047] In practical applications, due to the overlapping of structures such as scapula and rib in the lateral view whole body bone scan image, the lesion recognition difficulty is increased, the oblique view whole body bone scan image needs to be additionally calibrated in space coordinates, and the consistency of manual annotation cannot be guaranteed in the clinical practice due to the small amount of data collection in the non-standard view. Therefore, the two view whole body bone scan images in the embodiments of the present application adopt the anterior view whole body bone scan image Ant and the posterior view whole body bone scan image Post. The anterior-posterior acquisition mode can completely cover the whole body bone projection, so that the anterior view whole body bone scan image and the posterior view whole body bone scan image can be complementary, avoiding the lesion missed detection caused by tissue overlap (such as rib lesions being blocked by heart and lung tissue) in a single view, and the anatomical structures of the anterior view and the posterior view have natural symmetry, which can ensure the spatial position correspondence of the same lesion in different views, providing accurate spatial reference for feature fusion.

[0048] For example, using Philips Precedence SPECT / CT scanner, the patient lies supine on the SPECT / CT scan bed, and the anterior and posterior views are collected simultaneously by double probes, peak 140keV, window width 20%, scan speed 15-20cm / min, matrix 256*1024 for bone scan to obtain the anterior view whole body bone scan image and the posterior view whole body bone scan image.

[0049] Further, after obtaining the at least two view whole body bone scan images, the view whole body bone scan images can be preprocessed to improve the image quality of the view whole body bone scan images. The preprocessing can include cropping the blank area around the view whole body bone scan images, converting the view whole body bone scan images into grayscale images, extracting the region of interest based on the grayscale images (such as extracting a bone imaging adaptive rectangular range according to the pixel value being less than a first preset threshold (such as 230, etc.)), and adjusting the region of interest to a preset image size (such as 224*672, etc.).

[0050] In addition, in order to avoid the high concentration state of the bladder region during imaging due to the failure to empty the bladder before bone scan examination, the high concentration of the bladder region and the high concentration of the tumor have similar performance in the bone scan image, which will interfere with the bone lesion detection model. Therefore, after preprocessing the view whole body bone scan images, the concentration of the bladder region can be threshold segmented and filled to a second preset threshold (such as 220, etc.).

[0051] S20, extracting whole body bone scan features of each view whole body bone scan image through the feature extraction module, and extracting a clinical feature vector of the clinical information through the clinical information embedding module.

[0052] Specifically, the feature extraction module is used to extract features from the whole-body bone scan images. Since this application utilizes at least two whole-body bone scan images from different perspectives, the bone lesion detection model can include at least two feature extraction modules. The at least two feature extraction modules correspond one-to-one with the at least two whole-body bone scan images from different perspectives, with each feature extraction module being used to extract features from the whole-body bone scan image from its corresponding perspective. In other words, the at least two feature extraction modules perform feature extraction in parallel to increase the speed of whole-body bone scan feature extraction. Of course, in actual applications, a single feature extraction module can also be used to sequentially extract features from the whole-body bone scan images from each perspective.

[0053] In one embodiment, the feature extraction module includes N cascaded feature extraction units, and the N-3th feature extraction unit and the N-2th feature extraction unit are both jump-connected to the Nth feature extraction unit, so that the input items of the N-3th feature extraction unit, the N-2th feature extraction unit and the N-1th feature extraction unit are concatenated as the input items of the Nth feature extraction unit. For example, Figure 2 As shown in the figure, the feature extraction module consists of five feature extraction units, which are cascaded in sequence. An adder is set between the fourth feature extraction unit and the fifth feature extraction unit, and the second feature extraction unit and the third extraction unit are both connected to the adder, so that the second feature map extracted by the second feature extraction unit and the third feature map extracted by the third feature extraction unit are spliced ​​to the fourth feature map extracted by the fourth feature extraction unit along the channel direction through the adder, and the spliced ​​feature map is used as the input item of the fifth feature extraction unit, realizing the deep fusion of multi-scale feature maps, effectively alleviating the gradient vanishing problem of the deep network, avoiding the loss of features of small lesions (such as osteolytic lesions with a diameter of <5 mm) due to continuous downsampling, and improving the model's recognition ability for lesions of different sizes.

[0054] Based on this, the whole body bone scan features of the whole body bone scan image of each viewing angle are extracted by the feature extraction module specifically including:

[0055] For each viewing angle whole body bone scan image, the viewing angle whole body bone scan image is input into a first feature extraction unit, and a first feature map is extracted by the first feature extraction unit;

[0056] Input the first feature map into the second feature extraction unit, extract the second feature map through the second feature extraction unit, and so on, until the N-1th feature map is extracted through the N-1th feature extraction unit;

[0057] The (N-1)th feature map to the (N-M)th feature map are spliced, and the spliced feature map is input into the Nth feature extraction unit to extract the whole body bone scan feature of the view whole body bone scan image through the Nth feature extraction unit.

[0058] Specifically, N represents the number of feature extraction units included in the feature extraction module, and M represents the feature map data spliced to the (N-1)th feature map by the skip connection mode. Wherein, N is a positive integer greater than or equal to 5, M can be a positive integer determined according to N, and M is greater than 1, for example, M = N-2, etc.

[0059] In the embodiments of the present application, N = 5 and M = 3 are taken as examples for illustration. That is, the feature extraction module includes 5 feature extraction units, the feature maps output by the second, third and fourth feature extraction units are spliced in the channel splicing mode, and the spliced feature map is taken as the input item of the fifth feature extraction unit. Wherein, the first feature extraction unit is used to extract the global contour of the view whole body bone scan image to obtain the first feature map, the second feature extraction unit is used to refine the local structure of the view whole body bone scan image to obtain the second feature map, and the third to fifth feature extraction units are used to extract the global semantic.

[0060] For example, the first feature extraction unit uses an 11x11 convolution kernel (step 1, padding 5) to extract the global contour, and outputs a first feature map of 64 channels; the second feature extraction unit uses a 5x5 convolution kernel (step 1, padding 2) to refine the local structure, and outputs a second feature map of 128 channels; the third, fourth and fifth feature extraction units each include a convolution layer and an activation function layer, the convolution layer can use a 3x3 convolution kernel (step 1, padding 1), and the activation function layer can use a ReLU activation function to introduce a nonlinear mapping, and the third, fourth and fifth feature extraction units successively obtain a third feature map of 256 channels, a fourth feature map of 512 channels and a fifth feature map of 1024 channels. Wherein, the second feature map output by the second feature extraction unit and the third feature map output by the third feature extraction unit are fused with the fourth feature map output by the fourth feature extraction unit by channel splicing to form a multi-scale feature map, which retains the spatial details of the shallow network and the semantic information of the deep network, and avoids feature degradation of the deep network.

[0061] Further, after obtaining the fifth feature map extracted by the fifth feature extraction unit, the fifth feature map is converted into a feature vector (such as 1024 dimensions) through a flattening operation, and the feature vector is taken as an abstract representation of the view whole body bone scan image to obtain the whole body bone scan feature, and overfitting of the view whole body bone scan image caused by the deep network can be avoided.

[0062] The present application splices the second feature map (128 channels) and the third feature map (256 channels) to the fourth feature map (512 channels) through a skip connection mode, and realizes the deep fusion of multi-scale features by taking the spliced fourth feature map as the input of the fifth feature extraction unit. Among them, the shallow feature (i.e. the feature included in the second feature map) retains the edge contour and morphological details (such as the boundary features of a single condensation focus) of the lesion in the bone scan image, the middle layer feature (i.e. the feature contained in the third feature map) captures the spatial relationship (such as the positional relationship between the condensation focus and the joint and the vertebral body) of the lesion and the surrounding anatomical structure, and the deep feature (i.e. the feature contained in the fourth feature map) extracts the global semantic pattern (such as the distribution rule of multiple condensation focuses) of the lesion type, thereby forming a hierarchical feature expression of "local details-regional association-global semantics" in the fifth feature map, effectively alleviating the gradient vanishing problem of the deep network, avoiding the loss of features of small lesions (such as osteolytic lesions with a diameter of <5mm) caused by continuous downsampling, and improving the recognition ability of the model for lesions of different sizes. At the same time, when the extracted whole body bone scan features and the attention mechanism in the feature fusion module work together, the whole body bone scan features containing multi-scale features provided by the skip connection as the input of the attention mechanism can guide the bone lesion detection model to focus on the corresponding areas of the lesions in the anterior and posterior views (such as the exposed part of the lesion in the posterior view which is blocked in the anterior view), while reducing the repeated extraction of shallow features by the deep network, reducing the model parameter amount, and effectively avoiding overfitting.

[0063] Further, the clinical information includes gender, age, tumor history and other related information (such as symptom performance, laboratory test results, past medical history, etc.). The clinical information embedding module is used to convert the clinical information into a clinical feature vector, further enriching the feature dimension of the bone lesion detection model. Among them, the clinical information embedding module converts the clinical information into a calculable digital feature: gender and tumor history are processed as binary data of 0 / 1, age is converted to a value between 0-1 through normalization, forming a three-dimensional input vector. Then it is mapped to a 1024-dimensional feature vector through a linear transformation layer, which has the same dimension as the image feature, facilitating subsequent fusion. The clinical features and image features are fused by adding elements, when the patient has a tumor history, the model will enhance the attention to malignant lesion features and reduce the misjudgment of benign lesions (such as degenerative changes).

[0064] S30, fusing each whole body bone scan feature through the feature fusion module to obtain a fused feature vector.

[0065] Specifically, the feature fusion module is configured to fuse each whole body bone scan feature, and the feature fusion module adopts an attention mechanism to form a composite feature containing "single-view details" and "cross-view associations", thereby improving the ability to distinguish between malignant lesions such as bone metastases and benign lesions and effectively improving the accuracy of multi-classification diagnosis. The attention mechanism includes a self-attention mechanism and a cross-attention mechanism. The self-attention mechanism is configured to mine long-distance dependency relationships between foci and surrounding tissues in each view of the whole body bone scan image (i.e., single view), and the cross-attention mechanism is configured to realize information complementation across views (i.e., at least two views).

[0066] Further, the feature fusion module includes a scaled dot-product attention unit and a fusion unit. The scaled dot-product attention unit is configured to perform self-attention learning in each view of the whole body bone scan image and cross-attention learning between different views of the whole body bone scan image. The fusion unit is configured to fuse the self-attention feature vectors obtained through self-attention learning and the cross-attention feature vectors obtained through cross-attention learning to obtain a fusion feature vector containing "single-view details" and "cross-view associations".

[0067] Based on this, the feature fusion module includes a scaled dot-product attention unit and a fusion unit, and the fusion of each whole body bone scan feature through the feature fusion module to obtain a fusion feature vector specifically includes:

[0068] generating a query vector, a key vector, and a value vector based on each whole body bone scan feature;

[0069] determining a self-attention feature vector through the scaled dot-product attention unit based on the query vector, the key vector, and the value vector generated based on each whole body bone scan feature, and determining a cross-attention feature vector through the scaled dot-product attention unit based on the query vector generated based on each whole body bone scan feature and the key vector and the value vector generated based on each other whole body bone scan feature;

[0070] fusing all self-attention feature vectors and all cross-attention feature vectors through the fusion unit to obtain a fusion feature vector.

[0071] Specifically, each whole-body bone scan feature generates a query vector, a key vector and a value vector through a linear transformation Linear, inputs the query vector, the key vector and the value vector generated by each whole-body bone scan feature into a scaled dot-product attention unit, and determines the self-attention feature vector corresponding to each single-view whole-body bone scan image through the scaled dot-product attention unit. Then, for any one view whole-body bone scan image, the query vector of the view whole-body bone scan image and the key vector and the value vector generated by the other view whole-body bone scan images except the view whole-body bone scan image are input into the scaled dot-product attention unit, and all cross-attention feature vectors corresponding to each view whole-body bone scan image are output through the scaled dot-product attention unit. Further, as shown in Figure 2 The scaled dot-product attention unit (Scaled Dot-Product Attention) includes a first matrix multiplication layer, a scaling layer, a normalization layer and a second matrix multiplication layer; the first matrix multiplication layer, the scaling layer, the normalization layer and the second matrix multiplication layer are sequentially cascaded, the input of the first matrix multiplication layer is the query vector and the key vector, and the input of the second matrix multiplication layer is the output of the normalization layer and the value vector.

[0072] For example, the at least two view whole-body bone scan images include an anterior view whole-body bone scan image and a posterior view whole-body bone scan image, and the whole-body bone scan features include an anterior whole-body bone scan feature and a posterior whole-body bone scan feature. As shown in Figure 2 The query vector, the key vector and the value vector generated by the anterior whole-body bone scan feature are input into the scaled dot-product attention unit, and the self-attention feature vector of the anterior view is output through the scaled dot-product attention unit. The query vector, the key vector and the value vector generated by the posterior whole-body bone scan feature are input into the scaled dot-product attention unit, and the self-attention feature vector of the posterior view is output through the scaled dot-product attention unit. The query vector generated by the anterior whole-body bone scan feature and the key vector and the value vector generated by the posterior whole-body bone scan feature are input into the scaled dot-product attention unit, and the cross-attention feature vector of the anterior view-posterior view is output through the scaled dot-product attention unit. The query vector generated by the posterior whole-body bone scan feature and the key vector and the value vector generated by the anterior whole-body bone scan feature are input into the scaled dot-product attention unit, and the cross-attention feature vector of the posterior view- anterior view is output through the scaled dot-product attention unit.

[0073] Further, after obtaining the self-attention feature vectors and all cross-attention feature vectors of each view whole-body bone scan image, all the obtained self-attention feature vectors and cross-attention feature vectors are fused to obtain a fusion feature vector. Wherein, all the self-attention feature vectors and cross-attention feature vectors can be fused to obtain the fusion feature vector by using residual connection.

[0074] The long-distance dependent relationship between the focus and the surrounding tissue in the single-view whole-body bone scan image can be mined by self-attention learning, such as strengthening the spatial correlation of the focus of the spine and the rib, highlighting the shape of the focus of the scapula, and suppressing the irrelevant background interference of the high focus of the bladder. When the self-attention and the cross-attention are combined, on the one hand, the cross-attention realizes the complementary information between the views (such as the rib focus blocked by the heart in the forward direction can be identified through the backward exposed area), and on the other hand, the correlation weight of each whole-body bone scan feature is calculated, the spatial correspondence of the focus in different views is forced to be learned by the bone focus detection model, and the positioning deviation caused by the anatomic overlap is avoided. Then, all the self-attention feature vectors and the cross-attention feature vectors can be fused by using a residual connection to form a fusion feature vector containing “single-view details” and “cross-view correlation”, which improves the discrimination ability of malignant lesions such as bone metastases and benign lesions, and effectively improves the multi-classification prediction accuracy.

[0075] S40, determining a bone focus class based on the fusion feature vector and the clinical feature vector through the classification head.

[0076] Specifically, after obtaining the fusion feature vector and the clinical feature vector, the fusion feature vector and the clinical feature vector are spliced and input into the classification head, and the bone focus class is input through the classification head. Wherein, the bone focus class can include a benign class, a primary bone tumor class and a bone metastasis class, and then the classification head is a three-classification head; or, the bone focus class can include a normal class, a benign class, a primary bone tumor class and a bone metastasis class, and then the classification head is a four-classification head.

[0077] The normal class is:

[0078] (1) No 99m Tc-MDP focus in whole-body bone scan;

[0079] (2) No lesion in the bone region determined by CT, MRI examination.

[0080] The benign class is:

[0081] (1) 99m Tc-MDP focus appears in whole-body bone scan;

[0082] (2) Combined with CT, MRI, PET / CT and medical history to determine bone fracture, bone cyst, bone hyperplasia, bone bridge, degenerative osteoarthropathy, etc.

[0083] The primary bone tumor class is:

[0084] (1) There is obvious 99m Tc-MDP focus in whole-body bone scan;

[0085] (2) No history of malignant tumor;

[0086] (3) Pathological examination and laboratory examination determine primary malignant bone tumor such as osteosarcoma, chondrosarcoma, etc.

[0087] Bone metastasis categories are:

[0088] (1) Significant 99m Tc-MDP concentration foci appear in whole body bone scan;

[0089] (2) History of malignant tumors such as breast cancer, lung cancer, prostate cancer, etc.

[0090] (3) Pathological examination and laboratory examination determine bone metastasis.

[0091] In one embodiment, during the training phase of the bone lesion detection model, data augmentation methods such as brightness change (± 30%), contrast change (± 30%) and random horizontal flip are used to make the training images more diverse. Among them, the loss function adopts the Cross Entropy Loss function; the initial learning rate is set to 1e-4, and RAdam with weight decay of 0.01 is used as the optimizer. The learning rate is adjusted using the cosine annealing method with a period of 80, and the minimum learning rate is 5e-6. The batch size of the training is set to 40.

[0092] Specifically:

[0093] (1) Cross Entropy Loss

[0094] The Cross Entropy Loss function is used as the core loss calculation method for multi-classification tasks in the embodiments of the present application. Its essence is to measure the information entropy difference between the model prediction probability distribution and the real label distribution, and it is suitable for classification tasks of bone lesion types (such as normal, benign, primary malignant, bone metastasis). The formula of the Cross Entropy Loss function L is:

[0095]

[0096]

[0097] Where B is the number of batch samples, C is the number of classification categories (three or four categories), y i,c is the real label of sample i in category c (0 or 1), is the prediction probability of sample i belonging to category c. By minimizing this Cross Entropy Loss function, the model parameters can be optimized to make the prediction probability closer to the real distribution, especially suitable for the nonlinear differentiation of different lesion types in bone scan images.

[0098] (2) RAdam optimizer (Rectified Adam)

[0099] ​Optimizer selects RAdam, which can dynamically correct the bias of adaptive learning rate, solves the problem of large variance of Adam optimizer caused by insufficient sample size in the early training. Among them, in the early training, when the sample size t is less than the threshold t0, the correction factor β t Adjust the learning rate, β t The formula is:

[0100]

[0101] Where t0=4 / β2-2, β2 is the second moment decay factor (default 0.999), and through the correction, the learning rate fluctuation in early training can be avoided.

[0102] Adaptive momentum and variance update to maintain momentum m t And the exponential moving average of adaptive learning rate v t , where m t And v t The update process is:

[0103] m t = β1m t-1 +(1-β1)g t

[0104]

[0105] Where g t is the current gradient, β1 is the first moment decay factor (default 0.9), m t-1 is the momentum of the sample size t-1, and v t-1 is the adaptive learning rate of the sample size t-1.

[0106] By combining the root mean square gradient of RMSprop and the momentum of Adam, RAdam can more stably update the parameters in the process of bone scan image feature extraction, especially suitable for weight optimization of CNN network (such as 5-layer 2D-CNN), avoiding overfitting caused by monotonous bone scan image features.

[0107] (3) Cosine annealing learning rate scheduling (Cosine Annealing Learning Rate)

[0108] In order to optimize the training process, a cosine annealing strategy with a period of 80 is used to dynamically adjust the learning rate, and its formula is:

[0109]

[0110] Where η max =1e-6 is the initial learning rate, η min =5e-6 is the minimum learning rate, and T curT is the number of current training cycles max = 80 is the total number of cycles.

[0111] The cosine annealing learning rate scheduling can avoid local optimum and improve generalization ability. The cosine function periodically decays the learning rate, so that the model searches for the optimal solution with a smaller step size in the later training period, and jumps out of the local extreme point with a larger learning rate when the cycle is restarted. It is especially suitable for capturing subtle lesion features in bone scan images (such as weak differences in radioactive tracer concentration foci).

[0112] In one embodiment, to show that the prediction of the deep learning model is based on the lesion area in the image, the Grad-cam method is used to visualize the feature map output by the image feature extraction module in the bone lesion detection model, and the model focus area is displayed in the form of a heat map. Based on this, the method further comprises:

[0113] Obtaining a gradient map of the bone lesion category relative to each whole-body bone scan feature;

[0114] Determining the influence weight of each whole-body bone scan feature according to the gradient map, and linearly combining each whole-body bone scan feature based on the influence weight to obtain a combined feature map;

[0115] Extracting a positive contribution area in the combined feature map, and superimposing the positive contribution area on a perspective whole-body bone scan image to obtain a visual contribution map.

[0116] Specifically, the visual contribution map is used as a heat map of the focus of the bone lesion detection model. The gradient map is the gradient of the classification layer in the classification head relative to the whole-body bone scan feature; then the gradient is averaged in the spatial dimension to obtain the importance ranking of the whole-body bone scan feature in each perspective, and the influence weight is determined based on the importance ranking; finally, the whole-body bone scan features in each perspective are linearly combined based on the influence weight, the negative contribution area is filtered through the ReLU activation function, the key features are mapped to the heat map and superimposed on the original bone scan image. In this way, the positioning ability of the bone lesion detection model for the lesion can be clearly presented: the heat map of benign lesions focuses on single lesion (such as bone cyst area), the heat map of malignant lesions covers multiple concentration foci or irregular uptake areas, and the heat map of normal samples has no significant highlight area, thereby assisting doctors to verify whether the model makes decisions based on clinical true lesion features, and enhancing the explainability of the diagnosis process.

[0117] In addition, in order to illustrate the classification performance of the bone lesion detection model used in the embodiment of the present application, the embodiment of the present application also calculates Accuracy, Sensitivity, Specificity, F1-score, ROC curve and area under the curve AUC. The confusion matrix is ​​used to show the difference between the model prediction results and the true label. The mean difference in the age of patients in each category between the sample sets from two different sources was tested separately; the difference in gender distribution of each category between the sample sets from two different sources; the difference in AUC between the use of images alone and the use of images and clinical information after modeling, and the difference between the bone lesion detection model and the review results of two nuclear medicine doctors of different seniority. Finally, it was found that the statistical significance level of the method provided in the embodiment of the present application was p < 0.05.

[0118] In summary, this embodiment provides a bone lesion classification method based on whole-body bone scan images and clinical information, the method comprising acquiring whole-body bone scan images and clinical information from at least two perspectives; extracting whole-body bone scan features from each whole-body bone scan image through the feature extraction module, extracting clinical feature vectors of the clinical information through the clinical information embedding module; fusing the whole-body bone scan features through the feature fusion module to obtain a fused feature vector; and determining the bone lesion category through the classification head based on the fused feature vector and the clinical feature vector. By using whole-body bone scan images from at least two perspectives, the present application can avoid missed lesions due to tissue overlap in a single perspective, and the anatomical structures of the front and back perspectives have natural symmetry, which can ensure the spatial position correspondence of the same lesion in different perspectives, provide an accurate spatial reference for feature fusion, and improve the prediction accuracy of bone lesion categories. At the same time, the present application extracts a hierarchical feature expression of local details, regional associations, and global semantics through the feature extraction module, effectively alleviating the gradient vanishing problem of deep networks, avoiding the loss of small lesion features due to continuous downsampling, improving the model's ability to recognize lesions of different sizes, and thereby improving the prediction accuracy of bone lesion categories.

[0119] Based on the above-mentioned bone lesion classification method based on whole-body bone scan images and clinical information, this embodiment provides a bone lesion classification device based on whole-body bone scan images and clinical information, which uses a trained bone lesion detection model. The bone lesion detection model includes a feature extraction module, a feature fusion module, a classification head, and a clinical information embedding module; Figure 3 As shown, the device includes:

[0120] An acquisition module 100 is configured to acquire whole-body bone scan images and clinical information from at least two viewing angles;

[0121] The control module 200 is configured to extract whole-body bone scan features from the whole-body bone scan image at each viewing angle through the feature extraction module, extract a clinical feature vector of the clinical information through the clinical information embedding module, fuse the whole-body bone scan features through the feature fusion module to obtain a fused feature vector, and determine a bone lesion category based on the fused feature vector and the clinical feature vector through the classification head.

[0122] Based on the above-mentioned method for classifying bone lesions based on whole-body bone scan images and clinical information, this embodiment provides a computer-readable storage medium, which stores one or more programs. The one or more programs can be executed by one or more processors to implement the steps in the method for classifying bone lesions based on whole-body bone scan images and clinical information as described in the above-mentioned embodiment.

[0123] Based on the above bone lesion classification method based on whole-body bone scan images and clinical information, the present application also provides a terminal device, such as Figure 4 As shown, it includes at least one processor 20; a display screen 21; and a memory 22. It may also include a communications interface 23 and a bus 24. The processor 20, display screen 21, memory 22, and communications interface 23 can communicate with each other via bus 24. The display screen 21 is configured to display a preset user guidance interface in the initial setup mode. The communications interface 23 can transmit information. The processor 20 can call the logic instructions in the memory 22 to execute the method in the above embodiment.

[0124] In addition, the logic instructions in the memory 22 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product.

[0125] The memory 22, as a computer-readable storage medium, can be configured to store software programs or computer-executable programs, such as program instructions or modules corresponding to the methods in the embodiments of the present disclosure. The processor 20 executes the software programs, instructions, or modules stored in the memory 22 to perform functional applications and data processing, thereby implementing the methods in the above embodiments.

[0126] The memory 22 can include a program storage area and a data storage area, wherein the program storage area can store an operating system, application programs required by at least one function; the data storage area can store data created according to the use of the terminal device, etc. In addition, the memory 22 can include a high-speed random access memory, and can also include a non-volatile memory. For example, various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc. can also be a transitory storage medium.

[0127] In addition, the specific processes of the above-mentioned storage medium and the plurality of instruction processors in the terminal device load and execute have been described in detail in the above-mentioned method, and will not be described one by one here.

[0128] Finally, it should be pointed out that: the above examples are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A bone lesion classification method based on whole-body bone scan images and clinical information, characterized in that: Applying a trained bone lesion detection model, the bone lesion detection model includes a feature extraction module, a feature fusion module, a classification head, and a clinical information embedding module; the method includes: Acquire whole-body bone scan images and clinical information from at least two viewing angles; Extracting whole-body bone scan features of the whole-body bone scan image of each viewing angle by the feature extraction module, and extracting clinical feature vectors of the clinical information by the clinical information embedding module; The whole body bone scan features are fused by the feature fusion module to obtain a fused feature vector; The bone lesion category is determined by the classification head based on the fused feature vector and the clinical feature vector.

2. The bone lesion classification method based on whole-body bone scan images and clinical information according to claim 1, characterized in that: The feature extraction module includes N cascaded feature extraction units; the whole body bone scan features of each viewpoint whole body bone scan image extracted by the feature extraction module specifically include: For each viewing angle whole body bone scan image, the viewing angle whole body bone scan image is input into a first feature extraction unit, and a first feature map is extracted by the first feature extraction unit; Input the first feature map into the second feature extraction unit, extract the second feature map through the second feature extraction unit, and so on, until the N-1th feature map is extracted through the N-1th feature extraction unit; The N-1th feature map to the NMth feature map are spliced, and the spliced ​​feature maps are input into the Nth feature extraction unit, and the whole body bone scan features of the whole body bone scan image of the viewing angle are extracted by the Nth feature extraction unit.

3. The bone lesion classification method based on whole-body bone scan images and clinical information according to claim 1, characterized in that: The feature fusion module includes a scaled dot product attention unit and a fusion unit. The feature fusion module is used to fuse the features of the whole body bone scan to obtain a fused feature vector. Specifically, the feature fusion module includes: generating a query vector, a key vector, and a value vector based on each whole-body bone scan feature; Based on the query vector, key vector and value vector generated by each whole-body bone scan feature, a self-attention feature vector is determined by a scaled dot product attention unit; based on the query vector generated by each whole-body bone scan feature and the key vector and value vector generated by each other whole-body bone scan feature, a cross-attention feature vector is determined by a scaled dot product attention unit; All self-attention feature vectors and all cross-attention feature vectors are fused through the fusion unit to obtain a fused feature vector.

4. The bone lesion classification method based on whole-body bone scan images and clinical information according to claim 3, characterized in that: The scaled dot product attention unit includes a first matrix multiplication layer, a scaling layer, a normalization layer and a second matrix multiplication layer; the first matrix multiplication layer, the scaling layer, the normalization layer and the second matrix multiplication layer are cascaded in sequence, the input items of the first matrix multiplication layer are the query vector and the key vector, and the input items of the second matrix multiplication layer are the output items and the value vector of the normalization layer.

5. The bone lesion classification method based on whole-body bone scan images and clinical information according to any one of claims 1 to 4, characterized in that: The at least two-view whole-body bone scan images include an anterior-view whole-body bone scan image and a posterior-view whole-body bone scan image.

6. The bone lesion classification method based on whole-body bone scan images and clinical information according to claim 1, characterized in that: The method further comprises: Obtaining a gradient map of bone lesion categories relative to each whole-body bone scan feature; determining an influence weight of each whole-body bone scan feature according to the gradient map, and linearly combining each whole-body bone scan feature based on the influence weight to obtain a combined feature map; A positive contribution region in the combined feature map is extracted, and the positive contribution region is superimposed on a one-view whole-body bone scan image to obtain a visual contribution map.

7. A bone lesion classification device based on whole-body bone scan images and clinical information, characterized in that: A trained bone lesion detection model is applied, wherein the bone lesion detection model includes a feature extraction module, a feature fusion module, a classification head, and a clinical information embedding module; the device includes: an acquisition module, configured to acquire whole-body bone scan images and clinical information from at least two viewing angles; A control module is configured to extract whole-body bone scan features of the whole-body bone scan image at each viewing angle through the feature extraction module, extract a clinical feature vector of the clinical information through the clinical information embedding module; fuse the whole-body bone scan features through the feature fusion module to obtain a fused feature vector; and determine a bone lesion category based on the fused feature vector and the clinical feature vector through the classification head.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the bone lesion classification method based on whole-body bone scan images and clinical information as described in any one of claims 1-6.

9. A terminal device, characterized in that: include: processor and memory; The memory stores a computer-readable program executable by the processor; When the processor executes the computer-readable program, the processor implements the steps of the bone lesion classification method based on whole-body bone scan images and clinical information according to any one of claims 1 to 6.

Images