Method, apparatus, device and medium for processing bone X-ray images

Abstract

The present invention relates to the field of artificial intelligence technology, and provides a method, device, equipment and medium for processing bone X-ray images. It can use a network trained based on the Segment Anything Model to process bone X-ray images. The Segment Anything Model, as a visual foundation model, has powerful zero-shot learning ability. It can extract bone edge information related to lesions in the X-ray image under the prompt of the lesion bounding box, and use the bone edge map to guide the network to learn bone edge features, thereby improving the detection performance of orthopedic diseases, providing important clues for doctors to diagnose orthopedic diseases, and assisting doctors to improve the film reading speed and diagnostic accuracy.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and particularly to a method, device, equipment and medium for processing bone X-ray images. Background Art

[0002] With the aggravation of population aging, the incidence of orthopedic diseases is also rising continuously. Thanks to the characteristics of low cost, low radiation dose and easy operation of X-ray imaging technology, X-ray imaging technology has become the most commonly used imaging examination method for the preliminary diagnosis of orthopedic diseases. However, since X-ray imaging technology projects the three-dimensional human body structure onto a two-dimensional plane by using the penetration ability of X-rays, the bone X-ray image will inevitably have the problem of blurred bone edges caused by the overlap of bones and organs. However, the edges of bones in X-ray images can provide important information for doctors to diagnose orthopedic diseases. By observing the edges of bones, it is possible to judge whether there are abnormal conditions such as bone deformation and displacement.

[0003] In the prior art, for bone X-ray images, Ryu et al. used a detector and a classifier to simultaneously detect vertebral compression fractures and the severity of fractures in X-ray images, and Lee et al. used width and height attention mechanisms to improve the detector to achieve vertebral compression fracture detection. However, the above methods can only detect single-category orthopedic diseases and cannot fully exploit the bone edge information in X-ray images.

[0004] Therefore, in view of the above problems, it has become an urgent problem to study a method that can fully exploit the bone edge information in X-ray images to assist doctors in diagnosing orthopedic diseases. Summary of the Invention

[0005] In view of the above, it is necessary to provide a method, device, equipment and medium for processing bone X-ray images, aiming to solve the problem of inability to fully exploit the bone edge information in bone X-ray images.

[0006] A method for processing bone X-ray images, the method for processing bone X-ray images includes:

[0007] Constructing an initial network based on the Segment Anything Model, and constructing a target loss function; wherein, the initial network includes an image feature extraction module, a bone edge information mining module, a multi-scale feature fusion module, an edge branch, a classification branch and a regression branch; the target loss function includes an edge loss function and a detection loss function;

[0008] Obtaining historical bone X-ray images to construct a training set, and training the initial network based on the training set and the target loss function to obtain a target network;

[0009] In response to a processing instruction for a target skeletal X-ray image, the image feature extraction module is used to extract features from the target skeletal X-ray image to obtain target image features;

[0010] The multi-scale feature fusion module is used to fuse the target image features to obtain fused features;

[0011] The classification branch is used to perform classification processing on the fused features to obtain a target type, and the regression branch is used to perform position prediction on the fused features to obtain a target position;

[0012] A processing result for the target skeletal X-ray image is generated according to the target type and the target position.

[0013] According to a preferred embodiment of the present invention, the structure of the image feature extraction module is the same as the structure of the backbone part of the YoLov8 network;

[0014] The skeletal edge information mining module includes the Segment Anything model;

[0015] The structure of the multi-scale feature fusion module is the same as the structure of the neck of the YoLov8 network;

[0016] The edge branch includes a preset number of upsampling layers. The features output by each upsampling layer are concatenated with the corresponding features output by the multi-scale feature fusion module along the channel dimension, and the concatenated features are used as the input of the next upsampling layer; wherein, the preset number is the same as the number of levels of the target image features;

[0017] The structure of the classification branch and the structure of the regression branch are the same as the structure of the detection head of the YoLov8 network.

[0018] According to a preferred embodiment of the present invention, the method further includes:

[0019] During the process of training the initial network, for each sample in the training set, the image encoder of the Segment Anything model is used to extract the encoded image features of the sample;

[0020] Obtain the true bounding box of the lesion marked on the sample;

[0021] Using the true bounding box as a prompt, the bounding box prompt embedding of the sample is obtained by the prompt encoder of the Segment Anything model;

[0022] The encoded image features and the bounding box prompt embedding are input into the mask decoder of the Segment Anything model to obtain the skeletal mask of the sample;

[0023] An edge detector is used to perform edge detection on the bone mask to obtain an initial bone edge map related to the lesion.

[0024] According to a preferred embodiment of the present invention, after obtaining the initial bone edge map related to the lesion, the method further includes:

[0025] Obtain the target fusion feature corresponding to the sample output by the multi-scale feature fusion module;

[0026] Using the following formula, input the target fusion feature into the edge branch to predict the bone edge, and obtain the target bone edge map:

[0027] E M =P M ;

[0028]

[0029] E 0 =I edg ;

[0030] where E M represents the feature output by the M-th upsampling layer; P M represents the feature corresponding to the feature output by the M-th upsampling layer output by the multi-scale feature fusion module; E m-1 represents the feature output by the (m - 1)-th upsampling layer; E 0 represents the feature output by the 0-th upsampling layer; E m represents the feature output by the m-th upsampling layer; represents the upsampling operation on the feature output by the m-th upsampling layer; P m represents the feature corresponding to the feature output by the m-th upsampling layer output by the multi-scale feature fusion module; represents the concatenation operation of P m and ; Conv represents the convolution operation, and the size of the convolution kernel is 1×1; I edg represents the target bone edge map; m represents an integer between [1, M], and M is an integer greater than 1.

[0031] According to a preferred embodiment of the present invention, before constructing the target loss function, the method further includes:

[0032] Using the initial bone edge map output by the bone edge information mining module as a supervision signal to guide the edge branch to learn the bone edge information related to the lesion, and obtaining the edge loss function:

[0033]

[0034] where Ledg denotes the edge loss; N denotes the number of image pixels corresponding to each sample; g i denotes the i-th pixel value predicted by the edge branch; s i denotes the i-th pixel value of the initial bone edge map output by the bone edge information mining module; i is an integer greater than or equal to 0.

[0035] According to a preferred embodiment of the present invention, the construction of the target loss function includes:

[0036] The target loss function is constructed using the following formula:

[0037] L total = L det + L edg ;

[0038] wherein, L total denotes the target loss function; L det denotes the detection loss function, and the detection loss function is the same as the loss function of the YoLov8 network.

[0039] According to a preferred embodiment of the present invention, after generating the processing result of the target bone X-ray image according to the target type and the target position, the method further includes:

[0040] Marking the target bone X-ray image according to the processing result to obtain a marked image;

[0041] Sending the marked image to a specified terminal device.

[0042] A processing device for bone X-ray images, the processing device for bone X-ray images includes:

[0043] A construction unit, configured to construct an initial network based on a segment-everything model and construct a target loss function; wherein, the initial network includes an image feature extraction module, a bone edge information mining module, a multi-scale feature fusion module, an edge branch, a classification branch, and a regression branch; the target loss function includes an edge loss function and a detection loss function;

[0044] A training unit, configured to obtain historical bone X-ray images to construct a training set, and train the initial network based on the training set and the target loss function to obtain a target network;

[0045] An extraction unit, configured to, in response to a processing instruction for a target bone X-ray image, use the image feature extraction module to extract features from the target bone X-ray image to obtain target image features;

[0046] A fusion unit, configured to fuse the target image features by using the multi-scale feature fusion module to obtain fused features;

[0047] A processing unit, configured to classify the fused features by using the classification branch to obtain a target type, and predict the position of the fused features by using the regression branch to obtain a target position;

[0048] A generation unit, configured to generate a processing result of the target skeletal X-ray image according to the target type and the target position.

[0049] A computer device, comprising:

[0050] A memory, storing at least one instruction; and

[0051] A processor, configured to execute the instruction stored in the memory to implement the method for processing skeletal X-ray images.

[0052] A computer-readable storage medium, in which at least one instruction is stored, and the at least one instruction is executed by a processor in a computer device to implement the method for processing skeletal X-ray images.

[0053] It can be seen from the above technical solutions that the present invention uses a network trained based on the Segment Anything Model to process skeletal X-ray images. The Segment Anything Model, as a visual foundation model, has a powerful zero-shot learning ability. It can extract the skeletal edge information related to the lesion in the X-ray image under the prompt of the lesion bounding box, and use the skeletal edge map to guide the network to learn skeletal edge features, thereby improving the detection performance of orthopedic diseases, and can provide important clues for doctors to diagnose orthopedic diseases to assist doctors in improving the film reading speed and diagnosis accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 is a flowchart of a preferred embodiment of the method for processing skeletal X-ray images of the present invention.

[0055] Figure 2 is a functional module diagram of a preferred embodiment of the device for processing skeletal X-ray images of the present invention.

[0056] Figure 3 is a schematic structural diagram of a computer device of a preferred embodiment for implementing the method for processing skeletal X-ray images of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0057] In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0058] As Figure 1As shown, it is a flowchart of a preferred embodiment of the method for processing skeletal X-ray images of the present invention. According to different requirements, the order of steps in this flowchart can be changed, and some steps can be omitted.

[0059] The method for processing skeletal X-ray images is applied to one or more computer devices. The computer device is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions. Its hardware includes, but is not limited to, microprocessors, application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0060] The computer device can be any electronic product that can interact with users. For example, personal computers, tablets, smartphones, personal digital assistants (PDAs), game consoles, Internet Protocol Television (IPTV), smart wearable devices, etc.

[0061] The computer device may also include network devices and / or user devices. Among them, the network device includes, but is not limited to, a single network server, a server group composed of multiple network servers, or a cloud composed of a large number of hosts or network servers based on cloud computing (Cloud Computing).

[0062] The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), as well as big data and artificial intelligence platforms.

[0063] Among them, artificial intelligence (AI) is a theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to obtain the best results.

[0064] The basic technologies of artificial intelligence generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technologies, operation / interaction systems, and mechatronics. The software technologies of artificial intelligence mainly include several major directions such as computer vision technology, robotics, biometric technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0065] The network where the computer device is located includes but is not limited to the Internet, wide area network, metropolitan area network, local area network, virtual private network (VPN), etc.

[0066] S10, construct an initial network based on the Segment Anything Model (SAM), and construct a target loss function; wherein, the initial network includes an image feature extraction module, a bone edge information mining module, a multi-scale feature fusion module, an edge branch, a classification branch, and a regression branch; the target loss function includes an edge loss function and a detection loss function.

[0067] In this embodiment, the structure of the image feature extraction module is the same as the backbone part of the YoLov8 network;

[0068] The bone edge information mining module includes the Segment Anything Model;

[0069] The structure of the multi-scale feature fusion module is the same as the neck structure of the YoLov8 network;

[0070] The edge branch includes a preset number of upsampling layers, and the features output by each upsampling layer are concatenated with the corresponding features output by the multi-scale feature fusion module along the channel dimension, and the features obtained after concatenation are used as the input of the next upsampling layer; wherein, the preset number is the same as the number of levels of the features output by the image feature extraction module (that is, the preset number is the same as the number of levels of the target image features mentioned later);

[0071] The structure of the classification branch and the structure of the regression branch are the same as the detection head structure of the YoLov8 network.

[0072] S11, obtain historical bone X-ray images to construct a training set, and train the initial network based on the training set and the target loss function to obtain a target network.

[0073] In this embodiment, each sample in the training set has a marked ground truth bounding box of the lesion, which can be specifically marked manually or by other means.

[0074] In this embodiment, the initial network can be trained in the following manner to obtain the target network:

[0075] First, use the image feature extraction module to extract features from each sample in the training set.

[0076] For example: For each sample in the training set, that is, the skeletal X-ray image I, use I as the input of the image feature extraction module to learn image features. Specifically, the image feature extraction module can extract image features at M levels in total F m represents the learned image feature at the m-th level. The formula for the feature extraction process is as follows:

[0077]

[0078] where E n (I) represents the operation of extracting the image features of I.

[0079] Secondly, the lesions of orthopedic diseases usually manifest as bone deformation or dislocation of adjacent bones. By observing the edges of bones in X-ray images, it is possible to determine whether there are abnormalities in the bones, and thus diagnose orthopedic diseases. Therefore, the edge information of bones can be an important clue for doctors to analyze X-ray images and diagnose diseases. However, due to the projection characteristics of X-ray imaging, X-ray images usually have the problem of blurred edges, and manual annotation of bone edges is time-consuming, laborious, and costly. The Segment Anything Model, as a vision foundation model, has powerful zero-shot segmentation capabilities. This model can segment the area within the bounding box under the prompt of the bounding box, which makes it possible to fully mine the bone edge information related to the lesion from X-ray images. Therefore, in this embodiment, the bone edge information mining module, that is, the Segment Anything Model, is further used to mine the bone edge information of each sample.

[0080] Specifically, during the process of training the initial network, for each sample in the training set, use the image encoder of the Segment Anything Model to extract the encoded image features of the sample; obtain the true bounding box of the lesion marked on the sample; use the true bounding box as a prompt, and use the prompt encoder of the Segment Anything Model to obtain the bounding box prompt embedding of the sample; input the encoded image features and the bounding box prompt embedding into the mask decoder of the Segment Anything Model to obtain the bone mask of the sample; use an edge detector to perform edge detection on the bone mask to obtain an initial bone edge map related to the lesion.

[0081] For example: Using the X-ray image I as the input, use the image encoder of the Segment Anything Model to extract the X-ray image feature S I , and the formula is as follows:

[0082] S I = Φ(I);

[0083] Wherein, Φ(I) represents the extraction operation of the image encoder of the segmentation-all model on I.

[0084] Meanwhile, using the true bounding box B of the lesion in the X-ray image g as a prompt, the bounding box prompt embedding R is obtained by using the prompt encoder of the segmentation-all model g , and the formula is as follows:

[0085] R g = ξ(B g );

[0086] Wherein, ξ(B g ) represents the operation of the prompt encoder of the segmentation-all model on B g .

[0087] Then, the bounding box prompt embedding R g and the image feature S I are sent into the mask decoder of the segmentation-all model together to obtain the bone mask F within the lesion area mask , and the formula is as follows:

[0088] F mask = Ψ(R g , S I );

[0089] Wherein, Ψ(R g , S I ) represents the operation of the mask decoder of the segmentation-all model on R g and S I .

[0090] Finally, edge detection is performed on the bone mask F mask to capture the initial bone edge map I related to the lesion e , and the formula is as follows:

[0091] I e = D e (F mask );

[0092] Wherein, D e (F mask ) represents the edge detection operation of the edge detector on F mask , and can be specifically implemented by edge detection operators such as Canny.

[0093] Again, the multi-scale feature fusion module is used to perform multi-scale fusion on the features extracted by the image feature extraction module.

[0094] For example, the image features output by the image feature extraction module have different resolutions. Among them, the feature map with a larger resolution contains richer disease detail information, and the feature map with a smaller resolution pays more attention to the semantic information of the disease. Since there are many types of orthopedic diseases and their sizes vary, in order to enable the network to detect orthopedic diseases of different categories and sizes simultaneously, in this embodiment, the multi-scale feature fusion module is constructed to fully fuse the detail and semantic information. The formula for the feature fusion process is as follows:

[0095]

[0096] where represents the fusion operation of the multi-scale feature fusion module on ; represents the corresponding fused feature.

[0097] Again, use the edge branch to further guide the model to learn edge information.

[0098] Specifically, in order to enable the orthopedic disease auxiliary diagnosis network to have the ability to learn bone edge information, the edge branch is self-constructed, and the initial bone edge map output by the bone edge information mining module is used as the supervision signal to guide the edge branch to learn the bone edge information related to the lesion. Specifically, after obtaining the initial bone edge map related to the lesion, the method further includes:

[0099] Obtain the target fused feature corresponding to the sample output by the multi-scale feature fusion module;

[0100] Using the following formula, input the target fused feature into the edge branch to predict the bone edge, and obtain the target bone edge map:

[0101] E M = P M ;

[0102]

[0103] E 0 = I edg ;

[0104] where E M represents the feature output by the M-th upsampling layer; P M represents the feature corresponding to the feature output by the M-th upsampling layer output by the multi-scale feature fusion module; E m-1 represents the feature output by the (m - 1)-th upsampling layer; E 0 represents the feature output by the 0-th upsampling layer; E mDenote the features output by the m-th upsampling layer; Denote the upsampling operation on the features output by the m-th upsampling layer; P m Denote the features corresponding to the features output by the m-th upsampling layer and output by the multi-scale feature fusion module; Denote the concatenation operation on Pm and ; Conv denotes the convolution operation, and the size of the convolution kernel is 1×1; I edg Denote the target bone edge map; m represents an integer between [1, M], and M is an integer greater than 1.

[0105] For example: when M is 4, the features output by each upsampling layer and the corresponding features output by the multi-scale feature fusion module are concatenated along the channel dimension and used as the input of the next upsampling layer, and so on, gradually restoring the spatial resolution of the image and generating the final target bone edge map.

[0106] Finally, train the initial network based on the training set and the target loss function to obtain the target network.

[0107] Specifically, before constructing the target loss function, the method further includes:

[0108] Use the initial bone edge map output by the bone edge information mining module as a supervision signal to guide the edge branch to learn the bone edge information related to the lesion, and obtain the edge loss function:

[0109]

[0110] where, L edg Denote the edge loss; N denotes the number of image pixels corresponding to each sample; g i Denote the i-th pixel value predicted by the edge branch; s i Denote the i-th pixel value of the initial bone edge map output by the bone edge information mining module; i is an integer greater than or equal to 0.

[0111] In the training stage of the target network, use the initial bone edge map output by the bone edge information mining module to guide the edge branch to learn the edge information of the bone, and optimize the learning process of the edge information by constraining the consistency between the prediction result of the edge branch and the initial bone edge map, so that the prediction result of the edge branch is consistent with the initial bone edge map.

[0112] Further, the construction of the target loss function includes:

[0113] Construct the target loss function using the following formula:

[0114] L total= L det + L edg ;

[0115] Wherein, L total represents the target loss function; L det represents the detection loss function, and the detection loss function is the same as the loss function of the YoLov8 network.

[0116] It can be seen that during the training process, the loss function L total consists of two parts. Among them, the detection loss function L det is used to optimize the learning process of the classification branch and the regression branch, and the edge loss function L edg is used to optimize the learning process of the edge branch. Optimizing the edge branch, the classification branch, and the regression branch simultaneously can make the features learned by the network suitable for predicting the orthopedic disease category and location in the X-ray image and also suitable for predicting the bone edge.

[0117] In the above embodiment, the zero-shot segmentation ability of the vision-based model segmentation all models is used to improve the existing object detection network, and an edge branch auxiliary network is designed to fully mine the bone edge information, so that the trained target network can process the bone X-ray image more accurately.

[0118] S12. In response to a processing instruction for the target bone X-ray image, use the image feature extraction module to extract features from the target bone X-ray image to obtain target image features.

[0119] In this embodiment, the processing instruction can be triggered by relevant medical staff or automatically triggered when a bone X-ray image is detected to be uploaded.

[0120] S13. Use the multi-scale feature fusion module to fuse the target image features to obtain fused features.

[0121] In this embodiment, after feature fusion, detailed and semantic information can be fully fused to improve the detection accuracy.

[0122] S14. Use the classification branch to classify the fused features to obtain the target type, and use the regression branch to predict the location of the fused features to obtain the target location.

[0123] In this embodiment, after the target network is trained, when actually using the target network to process the skeletal X-ray image, only the image feature extraction module, the multi-scale feature fusion module, the classification branch, and the regression branch are used to complete the image processing, and finally the predicted orthopedic disease type and location are obtained. The skeletal edge information mining module and the edge branch are only used in the training stage to optimize each module of the entire network.

[0124] S15. Generate a processing result for the target skeletal X-ray image according to the target type and the target location.

[0125] In this embodiment, after generating the processing result for the target skeletal X-ray image according to the target type and the target location, the method further includes:

[0126] Mark the target skeletal X-ray image according to the processing result to obtain a marked image;

[0127] Send the marked image to a specified terminal device.

[0128] Among them, the specified terminal device can be the terminal device of relevant medical staff. For example: the orthopedic disease type and location predicted by the target network can be marked in the target skeletal X-ray image with special colors or boxes, etc., and the marked image can be sent to the doctor's office equipment to assist the doctor in quickly reviewing the film and making a diagnosis, which can not only improve the doctor's film review speed, but also improve the doctor's diagnosis efficiency.

[0129] It can be seen from the above technical solutions that the present invention can use a network trained based on the Segment Anything Model to process skeletal X-ray images. The Segment Anything Model, as a visual foundation model, has a powerful zero-shot learning ability. It can extract the skeletal edge information related to the lesion in the X-ray image under the prompt of the lesion bounding box, and use the skeletal edge map to guide the network to learn the skeletal edge features, thereby improving the detection performance of orthopedic diseases and providing important clues for doctors to diagnose orthopedic diseases to assist doctors in improving the film review speed and diagnosis accuracy.

[0130] As Figure 2 shown, it is a functional module diagram of a preferred embodiment of the processing device for skeletal X-ray images of the present invention. The processing device 11 for skeletal X-ray images includes a construction unit 110, a training unit 111, an extraction unit 112, a fusion unit 113, a processing unit 114, and a generation unit 115. The module / unit referred to in the present invention means a series of computer program segments that can be executed by a processor and can complete fixed functions, and are stored in a memory. In this embodiment, the functions of each module / unit will be described in detail in subsequent embodiments.

[0131] The building unit 110 is used to construct an initial network based on the Segment Anything Model and construct a target loss function. Among them, the initial network includes an image feature extraction module, a bone edge information mining module, a multi-scale feature fusion module, an edge branch, a classification branch, and a regression branch. The target loss function includes an edge loss function and a detection loss function.

[0132] The training unit 111 is used to obtain historical bone X-ray images to construct a training set, and train the initial network based on the training set and the target loss function to obtain a target network.

[0133] The extraction unit 112 is used to respond to a processing instruction for a target bone X-ray image, and use the image feature extraction module to extract features from the target bone X-ray image to obtain target image features.

[0134] The fusion unit 113 is used to use the multi-scale feature fusion module to fuse the target image features to obtain fused features.

[0135] The processing unit 114 is used to use the classification branch to perform classification processing on the fused features to obtain a target type, and use the regression branch to perform position prediction on the fused features to obtain a target position.

[0136] The generation unit 115 is used to generate a processing result for the target bone X-ray image according to the target type and the target position.

[0137] It can be seen from the above technical solutions that the present invention can use a network trained based on the Segment Anything Model to process bone X-ray images. The Segment Anything Model, as a visual foundation model, has a powerful zero-shot learning ability. It can extract bone edge information related to lesions in X-ray images under the prompt of lesion bounding boxes, and use the bone edge map to guide the network to learn bone edge features, thereby improving the detection performance of orthopedic diseases and providing important clues for doctors to diagnose orthopedic diseases, so as to assist doctors in improving the film reading speed and diagnosis accuracy.

[0138] As Figure 3 shown, it is a schematic structural diagram of a computer device of a preferred embodiment for implementing the method for processing bone X-ray images of the present invention.

[0139] The computer device 1 may include a memory 12, a processor 13, and a bus, and may also include a computer program stored in the memory 12 and executable on the processor 13, such as a program for processing bone X-ray images.

[0140] Those skilled in the art can understand that the schematic diagram is only an example of the computer device 1, and does not constitute a limitation on the computer device 1. The computer device 1 can be either a bus structure or a star structure. The computer device 1 can also include more or fewer other hardware or software than shown in the figure, or different component arrangements. For example, the computer device 1 can also include input / output devices, network access devices, etc.

[0141] It should be noted that the computer device 1 is only an example. Other existing or future electronic products that can be adapted to the present invention should also be included within the protection scope of the present invention and are hereby incorporated by reference.

[0142] Among them, the memory 12 includes at least one type of readable storage medium. The readable storage medium includes flash memory, mobile hard disk, multimedia card, card-type memory (such as SD or DX memory, etc.), magnetic memory, magnetic disk, optical disc, etc. The memory 12 can be an internal storage unit of the computer device 1 in some embodiments, such as the mobile hard disk of the computer device 1. The memory 12 can also be an external storage device of the computer device 1 in other embodiments, such as a plug-in mobile hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. equipped on the computer device 1. Further, the memory 12 can also include both an internal storage unit and an external storage device of the computer device 1. The memory 12 can be used not only to store application software installed on the computer device 1 and various types of data, such as the code of the processing program for skeletal X-ray images, etc., but also to temporarily store data that has been output or will be output.

[0143] The processor 13 can be composed of integrated circuits in some embodiments. For example, it can be composed of a single packaged integrated circuit, or can be composed of multiple integrated circuits with the same or different functions packaged together, including a combination of one or more Central Processing Units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 13 is the control core (Control Unit) of the computer device 1, connecting various components of the entire computer device 1 through various interfaces and circuits. By running or executing programs or modules stored in the memory 12 (such as executing the processing program for skeletal X-ray images, etc.), and calling data stored in the memory 12, it performs various functions of the computer device 1 and processes data.

[0144] The processor 13 executes the operating system of the computer device 1 and various installed application programs. The processor 13 executes the application programs to implement the steps in the above-mentioned embodiments of the processing method for each skeletal X-ray image. For example Figure 1 the steps shown.

[0145] Exemplarily, the computer program can be divided into one or more modules / units. The one or more modules / units are stored in the memory 12 and executed by the processor 13 to complete the present invention. The one or more modules / units can be a series of computer-readable instruction segments capable of performing specific functions, and these instruction segments are used to describe the execution process of the computer program in the computer device 1. For example, the computer program can be divided into a construction unit 110, a training unit 111, an extraction unit 112, a fusion unit 113, a processing unit 114, and a generation unit 115.

[0146] The integrated units implemented in the form of software function modules can be stored in a computer-readable storage medium. The above-mentioned software function modules are stored in a storage medium, including several instructions for causing a computer device (which can be a personal computer, a computer device, or a network device, etc.) or a processor to execute a part of the processing method for skeletal X-ray images in various embodiments of the present invention.

[0147] If the modules / units integrated in the computer device 1 are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, to implement all or part of the processes in the above-mentioned embodiment methods of the present invention, it can also be completed by a computer program instructing relevant hardware devices. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0148] Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file, or some intermediate form, etc. The computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disc, a computer memory, a read-only memory (ROM, Read-Only Memory), a random access memory, etc.

[0149] Furthermore, the computer-readable storage medium mainly includes a program storage area and a data storage area. Among them, the program storage area can store an operating system, application programs required for at least one function, etc.; the data storage area can store data created according to the use of blockchain nodes, etc.

[0150] The blockchain referred to in the present invention is a new application mode of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, and encryption algorithm. Blockchain, in essence, is a decentralized database, a string of data blocks generated by using cryptographic methods. Each data block contains information about a batch of network transactions, which is used to verify the validity (anti-counterfeiting) of the information and generate the next block. The blockchain can include a blockchain underlying platform, a platform product service layer, an application service layer, etc.

[0151] The bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of convenience in representation, in Figure 3 it is only represented by a single straight line, but it does not mean that there is only one bus or one type of bus. The bus is arranged to realize the connection and communication between the memory 12 and at least one processor 13, etc.

[0152] Although not shown, the computer device 1 may further include a power supply (such as a battery) for powering each component. Preferably, the power supply can be logically connected to the at least one processor 13 through a power management device, so as to realize functions such as charge management, discharge management, and power consumption management through the power management device. The power supply may further include any components such as one or more DC or AC power supplies, a recharge device, a power failure detection circuit, a power converter or inverter, and a power status indicator. The computer device 1 may further include various sensors, a Bluetooth module, a Wi-Fi module, etc., which will not be elaborated here.

[0153] Furthermore, the computer device 1 may further include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is usually used to establish a communication connection between the computer device 1 and other computer devices.

[0154] Optionally, the computer device 1 may further include a user interface, which may be a display, an input unit (such as a keyboard), and optionally, the user interface may also be a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch liquid crystal display, and an OLED (Organic Light-Emitting Diode) toucher, etc. Among them, the display may also be appropriately referred to as a display screen or a display unit, which is used to display the information processed in the computer device 1 and to display a visual user interface.

[0155] It should be understood that the above embodiments are only for illustration purposes and are not limited by this structure in the scope of the patent application.

[0156] Those skilled in the art can understand that Figure 3 the structure shown does not constitute a limitation on the computer device 1, and it may include fewer or more components than shown in the figure, or combine some components, or have different component arrangements.

[0157] In combination with Figure 1 , the memory 12 in the computer device 1 stores a plurality of instructions to implement a method for processing skeletal X-ray images, and the processor 13 can execute the plurality of instructions to implement:

[0158] Construct an initial network based on the segmentation everything model and construct a target loss function; wherein, the initial network includes an image feature extraction module, a skeletal edge information mining module, a multi-scale feature fusion module, an edge branch, a classification branch, and a regression branch; the target loss function includes an edge loss function and a detection loss function;

[0159] Obtain historical skeletal X-ray images to construct a training set, and train the initial network based on the training set and the target loss function to obtain a target network;

[0160] In response to a processing instruction for a target skeletal X-ray image, use the image feature extraction module to extract features from the target skeletal X-ray image to obtain target image features;

[0161] Use the multi-scale feature fusion module to fuse the target image features to obtain fused features;

[0162] Use the classification branch to perform classification processing on the fused features to obtain a target type, and use the regression branch to perform position prediction on the fused features to obtain a target position;

[0163] Generate a processing result for the target bone X-ray image according to the target type and the target location.

[0164] Specifically, for the specific implementation method of the above instructions by the processor 13, reference may be made to Figure 1 the description of the relevant steps in the corresponding embodiment, which will not be elaborated here.

[0165] It should be noted that all the data involved in this case are legally obtained.

[0166] In several embodiments provided by the present invention, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

[0167] The present invention can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multi-processor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and so on. The present invention can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0168] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they can be located in one place, or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0169] In addition, in each embodiment of the present invention, the various functional modules can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above integrated unit can be implemented in the form of hardware, or in the form of a combination of hardware and software functional modules.

[0170] For those skilled in the art, it is obvious that the present invention is not limited to the details of the above-described exemplary embodiments, and the present invention can be implemented in other specific forms without departing from the spirit or basic characteristics of the present invention.

[0171] Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting. The scope of the present invention is defined by the appended claims rather than the above description. Therefore, all changes falling within the meaning and scope of the equivalent elements of the claims are intended to be embraced within the present invention. Any reference signs in the claims should not be construed as limiting the claims involved.

[0172] In addition, it is obvious that the term "comprising" does not exclude other units or steps, and the singular does not exclude the plural. The multiple units or devices described in the present invention can also be implemented by one unit or device through software or hardware. The terms such as "first" and "second" are used to denote names and do not denote any particular order.

[0173] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be modified or equivalently replaced without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for processing a skeletal X-ray image, characterized in that, The method for processing the skeletal X-ray image includes: Constructing an initial network based on the Segment Anything model and constructing a target loss function; wherein, the initial network includes an image feature extraction module, a skeletal edge information mining module, a multi-scale feature fusion module, an edge branch, a classification branch, and a regression branch; the target loss function includes an edge loss function and a detection loss function; Obtaining historical skeletal X-ray images to construct a training set, and training the initial network based on the training set and the target loss function to obtain a target network; In response to a processing instruction for a target skeletal X-ray image, using the image feature extraction module to extract features from the target skeletal X-ray image to obtain target image features; Using the multi-scale feature fusion module to fuse the target image features to obtain fused features; Using the classification branch to perform classification processing on the fused features to obtain a target type, and using the regression branch to perform position prediction on the fused features to obtain a target position; Generating a processing result for the target skeletal X-ray image according to the target type and the target position; Wherein, during the process of training the initial network, for each sample in the training set, using the image encoder of the Segment Anything model to extract the encoded image features of the sample; obtaining the true bounding box of the lesion marked on the sample; using the true bounding box as a prompt, using the prompt encoder of the Segment Anything model to obtain the bounding box prompt embedding of the sample; inputting the encoded image features and the bounding box prompt embedding into the mask decoder of the Segment Anything model to obtain the skeletal mask of the sample; using an edge detector to perform edge detection on the skeletal mask to obtain an initial skeletal edge map related to the lesion; Wherein, after obtaining the initial skeletal edge map related to the lesion, obtaining the target fused features corresponding to the sample output by the multi-scale feature fusion module; Using the following formula, inputting the target fused features into the edge branch to predict the skeletal edge to obtain a target skeletal edge map: ； ； ； Among them, represents the feature output by the M-th upsampling layer; represents the feature corresponding to the feature output by the M-th upsampling layer output by the multi-scale feature fusion module; represents the feature output by the (m-1)-th upsampling layer; represents the feature output by the 0-th upsampling layer; represents the feature output by the m-th upsampling layer; represents the upsampling operation on the feature output by the m-th upsampling layer; represents the feature corresponding to the feature output by the m-th upsampling layer output by the multi-scale feature fusion module; represents the concatenation operation on and ; represents a convolution operation, and the size of the convolution kernel is ; represents the target bone edge map; m represents an integer between [1, M], and M is an integer greater than 1; Wherein, before constructing the target loss function, using the initial skeletal edge map output by the skeletal edge information mining module as a supervision signal to guide the edge branch to learn the skeletal edge information related to the lesion to obtain the edge loss function: ； Among them, represents the edge loss; represents the number of image pixels corresponding to each sample; represents the th pixel value predicted by the edge branch; represents the th pixel value of the initial bone edge map output by the bone edge information mining module; i is an integer greater than or equal to 0.

2. The method for processing a skeletal X-ray image according to claim 1, wherein: The structure of the image feature extraction module is the same as the backbone part of the YoLov8 network; The skeletal edge information mining module includes the Segment Anything model; The structure of the multi-scale feature fusion module is the same as the neck structure of the YoLov8 network; The edge branch includes a preset number of upsampling layers, and the features output by each upsampling layer are concatenated with the corresponding features output by the multi-scale feature fusion module along the channel dimension, and the concatenated features are used as the input of the next upsampling layer; wherein, the preset number is the same as the number of levels of the target image features; The structures of the classification branch and the regression branch are the same as those of the detection head of the YoLov8 network.

3. The processing method of the skeletal X-ray image according to claim 2, characterized in that, The construction of the target loss function includes: The target loss function is constructed using the following formula: ； Among them, represents the target loss function; represents the detection loss function, and the detection loss function is the same as the loss function of the YoLov8 network.

4. The method for processing a skeletal X-ray image according to claim 1, wherein, After generating the processing result of the target skeletal X-ray image according to the target type and the target position, the method further includes: Marking the target skeletal X-ray image according to the processing result to obtain a marked image; Sending the marked image to a specified terminal device.

5. A processing device for skeletal X-ray images, characterized in that, The processing device for the skeletal X-ray image includes: A construction unit, configured to construct an initial network based on the Segment Anything model and construct a target loss function; wherein, the initial network includes an image feature extraction module, a skeletal edge information mining module, a multi-scale feature fusion module, an edge branch, a classification branch, and a regression branch; the target loss function includes an edge loss function and a detection loss function; A training unit, configured to obtain historical skeletal X-ray images to construct a training set, and train the initial network based on the training set and the target loss function to obtain a target network; An extraction unit, configured to, in response to a processing instruction for a target skeletal X-ray image, extract features of the target skeletal X-ray image using the image feature extraction module to obtain target image features; A fusion unit, configured to fuse the target image features using the multi-scale feature fusion module to obtain fused features; A processing unit, configured to classify the fused features using the classification branch to obtain a target type, and predict the position of the fused features using the regression branch to obtain a target position; A generation unit, configured to generate a processing result for the target skeletal X-ray image according to the target type and the target position; Wherein, during the process of training the initial network, for each sample in the training set, the encoded image features of the sample are extracted using the image encoder of the Segment Anything model; the true bounding box of the lesion marked on the sample is obtained; using the true bounding box as a prompt, the bounding box prompt embedding of the sample is obtained using the prompt encoder of the Segment Anything model; the encoded image features and the bounding box prompt embedding are input into the mask decoder of the Segment Anything model to obtain the skeletal mask of the sample; the skeletal mask is edge-detected using an edge detector to obtain an initial skeletal edge map related to the lesion; Wherein, after obtaining the initial skeletal edge map related to the lesion, the target fused features corresponding to the sample output by the multi-scale feature fusion module are obtained; Using the following formula, the target fused features are input into the edge branch to predict the skeletal edge to obtain a target skeletal edge map: ； ； ； Among them, represents the feature output by the M-th upsampling layer; represents the feature corresponding to the feature output by the M-th upsampling layer output by the multi-scale feature fusion module; represents the feature output by the (m-1)-th upsampling layer; represents the feature output by the 0-th upsampling layer; represents the feature output by the m-th upsampling layer; represents the upsampling operation on the feature output by the m-th upsampling layer; represents the feature corresponding to the feature output by the m-th upsampling layer output by the multi-scale feature fusion module; represents and the concatenation operation; represents a convolution operation, and the size of the convolution kernel is ; represents the target bone edge map; m represents an integer between [1, M], and M is an integer greater than 1; Wherein, before constructing the target loss function, the initial skeletal edge map output by the skeletal edge information mining module is used as a supervision signal to guide the edge branch to learn the skeletal edge information related to the lesion to obtain the edge loss function: ； Among them, represents the edge loss; represents the number of image pixels corresponding to each sample; represents the th pixel value predicted by the edge branch; represents the th pixel value of the initial bone edge map output by the bone edge information mining module; i is an integer greater than or equal to 0.

6. A computer device, characterized in that, The computer device includes: A memory, storing at least one instruction; and A processor that executes instructions stored in the memory to implement the method for processing skeletal X-ray images according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that: At least one instruction is stored in the computer-readable storage medium, and the at least one instruction is executed by a processor in a computer device to implement the method for processing skeletal X-ray images according to any one of claims 1 to 4.

Images