Hand bone image analysis method based on feature extraction and related equipment

Abstract

The present application provides a hand bone image analysis method and related equipment based on feature extraction. First, a trained main neural network extracts the global features of the patient's hand bone image to obtain a global feature vector, and then extracts the key area features by selecting the key area to obtain the key area feature vector, and then splices the global feature vector and the key area feature vector to obtain a splicing vector; the trained main neural network diagnoses the global feature vector, the key area feature vector and the splicing vector respectively to obtain a global diagnosis result, a key area diagnosis result and a splicing diagnosis result, and then performs a comprehensive diagnosis on the hand bone image based on the above three diagnosis results to determine whether the patient suffers from Kashin-Beck disease. The method provided in the present application ensures the efficiency and accuracy of Kashin-Beck disease diagnosis, meets the needs of large-scale screening of patients with Kashin-Beck disease, and solves the problem of low efficiency in manually judging whether an image is sick during large-scale screening in related technologies.

Description

Technical Field

[0001] The present application relates to the field of neural network technology, and in particular to a hand bone image analysis method based on feature extraction and related equipment. Background Art

[0002] Kaschin-Beck disease is an endemic, deforming joint bone disease, also known as dwarfism and abacus disease in China. It is widely distributed in China, mostly in Heilongjiang, Jilin, Liaoning, Shanxi, Shaanxi and other provinces. It is also common in mountainous and semi-mountainous areas, and rare in plains. Most patients are children and adolescents, with clinical symptoms such as joint pain, deformity, dwarfism, and movement disorders.

[0003] The X-ray images of KBD in different situations (different ages, locations, and degrees of lesions in patients) usually show different manifestations. Among them, the early stage of the patient's condition can often be reflected in the hand X-ray images. Therefore, patients can be screened by large-scale inspection of hand X-ray images in prone areas, so as to detect patients as early as possible and provide timely treatment.

[0004] For large-scale screening of patients with KBD, the speed of manual identification of image features is relatively slow, and a KBD diagnosis method is needed that can efficiently meet the needs of large-scale screening. Summary of the invention

[0005] In view of this, the purpose of this application is to propose a hand bone image analysis method based on feature extraction and related equipment.

[0006] Based on the above purpose, the present application provides a hand bone image analysis method based on feature extraction, comprising: preprocessing the initial hand bone image of the patient to obtain a hand bone image; extracting and obtaining the global features of the hand bone image from a main neural network, and generating a global feature vector of the hand bone image, the main neural network performing a global diagnosis on the global feature vector to obtain a global diagnosis result, wherein the main neural network is obtained by training the initial main neural network using the collected data set; selecting a plurality of key areas of the hand bone image from a region selection neural network to obtain a plurality of key area images, wherein the region selection neural network is obtained by training the initial region selection neural network using the collected data set; the main The neural network extracts features of multiple key area images to obtain key area features of multiple hand bone images and generates multiple key area feature vectors of the hand bone images. The main neural network performs key area diagnosis on the multiple key area feature vectors to obtain multiple key area diagnosis results. The main neural network splices the global feature vector with the key area feature vector to obtain a splicing vector. The main neural network performs splicing diagnosis on the splicing vector to obtain a splicing diagnosis result. The main neural network performs a comprehensive diagnosis on the hand bone images based on the global diagnosis result, the key area diagnosis result and the splicing diagnosis result to obtain a final diagnosis result of whether the patient suffers from Kaschin-Beck disease.

[0007] Optionally, the method also includes: constructing the initial main neural network and the initial region selection neural network; acquiring the data set, the data set including multiple initial sample hand bone images, diagnosis results of manually performing comprehensive diagnosis on the multiple initial sample hand bone images, and coordinates of key regions manually selected on the multiple initial sample hand bone images; training the initial main neural network based on the data set to obtain the main neural network; training the initial region selection neural network based on the data set to obtain the region selection neural network.

[0008] Optionally, the initial main neural network is trained based on the data set to obtain the main neural network, including: obtaining the initial sample hand bone image from the data set, preprocessing the initial sample hand bone image to obtain a sample hand bone image; the initial main neural network performs global diagnosis processing on the sample hand bone image to obtain a sample global diagnosis result, and uses the sample global diagnosis result to calculate the global loss L g The initial main neural network performs key area diagnosis processing on the sample hand bone image to obtain multiple sample key area diagnosis results, and uses the multiple sample key area diagnosis results to calculate the key area loss L pThe initial main neural network performs splicing diagnosis processing on the sample hand bone images to obtain multiple sample splicing diagnosis results, and uses the multiple sample splicing diagnosis results to calculate the splicing loss L * ; The initial main neural network uses the global loss L g , the critical area loss L p And the splicing loss L * Calculate the overall network loss L total ; Based on the overall network loss L total Adjust the parameters of the initial main neural network to complete a training of the initial main neural network; in response to determining the overall network loss L total All the data sets that are less than or equal to a preset threshold or the initial main neural network complete training, and the initial main neural network after training is used as the main neural network.

[0009] Optionally, the method further includes: the initial main neural network performs global diagnostic processing on the sample hand bone image to obtain a sample global diagnostic result, including: the initial main neural network extracts and obtains the sample global features of the sample hand bone image, and generates a sample global feature vector of the sample hand bone image, and the initial main neural network performs global diagnosis on the sample global feature vector to obtain the sample global diagnostic result; the initial main neural network performs key area diagnostic processing on the sample hand bone image to obtain multiple sample key area diagnostic results, including: the initial area selection neural network selects multiple key areas of the sample hand bone image to obtain multiple sample key area images; the initial main neural network The network extracts features of multiple sample key area images to obtain sample key area features of multiple sample hand bone images, and generates multiple sample key area feature vectors of the sample hand bone images; the initial main neural network performs key area diagnosis on the multiple sample key area feature vectors to obtain multiple sample key area diagnosis results; the initial main neural network performs splicing diagnosis processing on the sample hand bone images to obtain multiple sample splicing diagnosis results, including: the initial main neural network splices the sample global feature vector with the sample key area feature vector to obtain a sample splicing vector; the initial main neural network performs splicing diagnosis on the sample splicing vector to obtain the sample splicing diagnosis result.

[0010] Optionally, the initial main neural network uses the global loss L g , the critical area loss L p And the splicing loss L * Calculate the overall network loss L total ,include:

[0011] The initial main neural network uses the following formula to calculate the overall network loss L total :

[0012] L total =L * +λL g +βL p

[0013] The parameters λ and β are the global loss L g With the critical area loss L p The weight of .

[0014] Optionally, the main neural network performs a comprehensive diagnosis on the hand bone image according to the global diagnosis result, the key area diagnosis result and the splicing diagnosis result to obtain a final diagnosis result of whether the patient suffers from KBD, including: the main neural network assigns different weights to the global diagnosis result, the key area diagnosis result and the splicing diagnosis result, and obtains the result value through calculation; the main neural network obtains the final diagnosis result by comparing the result value with a preset diagnosis threshold.

[0015] Optionally, the global features include the distribution of hand bones in the hand bone image and the texture of the bones; the key areas include the epiphysis of the middle and base phalanges of the ring finger, middle finger and index finger, the epiphysis of the distal phalanx of the thumb, and the proximal end of the third metacarpal bone; the key area features include depression, sclerosis, and unevenness of the epiphysis.

[0016] Based on the above purpose, the present application also provides a hand bone image analysis device based on feature extraction, including: a preprocessing module, configured to preprocess the initial hand bone image of the patient to obtain a hand bone image; a global diagnosis module, configured to extract and obtain the global features of the hand bone image for the main neural network, and generate a global feature vector of the hand bone image, and the main neural network performs a global diagnosis on the global feature vector to obtain a global diagnosis result; a key area selection module, configured to select multiple key areas of the hand bone image by the area selection neural network to obtain multiple key area images; the key area diagnosis module is configured to extract the features of the multiple key area images by the main neural network to obtain multiple The key area features of the hand bone image are obtained by generating a plurality of key area feature vectors of the hand bone image, the main neural network performs key area diagnosis on the plurality of key area feature vectors, and obtains a plurality of key area diagnosis results; the splicing diagnosis module is configured as the main neural network splicing the global feature vector with the key area feature vector to obtain a splicing vector, the main neural network performs splicing diagnosis on the splicing vector to obtain a splicing diagnosis result; the final result diagnosis module is configured as the main neural network performing a comprehensive diagnosis on the hand bone image according to the global diagnosis result, the key area diagnosis result and the splicing diagnosis result, to obtain a final diagnosis result of whether the patient suffers from KBD.

[0017] Based on the above purpose, the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes any one of the hand bone image analysis methods based on feature extraction.

[0018] Based on the above purpose, the present application also provides a non-transitory computer-readable storage medium, which stores computer instructions, characterized in that the computer instructions are used to enable a computer to execute any one of the hand bone image analysis methods based on feature extraction.

[0019] From the above, it can be seen that the present application provides a method for analyzing hand bone images based on feature extraction and related equipment. First, the trained main neural network extracts the global features of the patient's hand bone images to obtain a global feature vector, and then extracts the key area features by selecting the key area to obtain the key area feature vector, and then splices the global feature vector and the key area feature vector to obtain a splicing vector; the trained main neural network diagnoses the global feature vector, the key area feature vector and the splicing vector respectively to obtain a global diagnosis result, a key area diagnosis result and a splicing diagnosis result, and then performs a comprehensive diagnosis on the hand bone image based on the above three diagnosis results to determine whether the patient suffers from Kashin-Beck disease. The method provided in the present application ensures the efficiency and accuracy of the diagnosis of Kashin-Beck disease, meets the needs of large-scale screening of patients with Kashin-Beck disease, and solves the problem of low efficiency in manually judging whether an image is sick during large-scale screening in related technologies. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the present application or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 A flowchart of a hand bone image analysis method based on feature extraction according to an embodiment of the present application;

[0022] Figure 2 This is a flowchart of the sample hand bone image preprocessing process of the embodiment of the present application;

[0023] Figure 3 A flowchart of global diagnosis in an embodiment of the present application;

[0024] Figure 4 A flowchart of the key area selection and key area diagnosis process of the embodiment of the present application;

[0025] Figure 5 A flowchart of splicing diagnosis in an embodiment of the present application;

[0026] Figure 6 This is a flowchart of the initial main neural network training process of the embodiment of the present application;

[0027] Figure 7 A block diagram of a hand bone image analysis device based on feature extraction according to an embodiment of the present application;

[0028] Figure 8 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0029] In order to make the objectives, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0030] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present application should be the usual meanings understood by people with ordinary skills in the field to which the present application belongs. The "first", "second" and similar words used in the embodiments of the present application do not represent any order, quantity or importance, but are only used to distinguish different components. "Including" or "comprising" and similar words mean that the elements or objects appearing in front of the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0031] For ease of understanding, the terms involved in the embodiments of the present disclosure are explained below:

[0032] ResNet: Residual Network, which is easy to optimize and can improve accuracy by increasing the depth. Its internal residual blocks use skip connections to alleviate the gradient vanishing problem caused by increasing the depth in deep neural networks.

[0033] ResNeXt: A network obtained by combining a residual network with a group convolution strategy;

[0034] F1 index: A comprehensive evaluation based on precision and recall in information retrieval and natural language processing (i.e., the harmonic mean of the two).

[0035] As described in the background technology, for large-scale screening of patients with KBD, the speed of manual identification of image features is relatively slow. A KBD diagnosis method is needed that can efficiently meet the needs of large-scale screening. Applying neural networks to the field of medical artificial intelligence and automatically extracting image features can greatly improve the efficiency and accuracy of diagnosis.

[0036] In view of this, the present invention proposes a hand bone image analysis method based on feature extraction using deep learning neural network technology. The neural network automatically extracts the features of the overall image and common lesion areas, performs feature splicing, and uses the spliced ​​features to achieve feature expression with higher coverage of the image.

[0037] For ease of understanding, the terms involved in the embodiments of the present disclosure are explained below:

[0038] In view of this, an embodiment of the present application provides a hand bone image analysis method based on feature extraction, such as Figure 1 As shown, including:

[0039] S101, the initial hand bone image of the patient is preprocessed to obtain a hand bone image.

[0040] S102, the main neural network extracts and obtains the global features of the hand bone image, and generates a global feature vector of the hand bone image, and the main neural network performs a global diagnosis on the global feature vector to obtain a global diagnosis result, wherein the main neural network is an initial main neural network trained using the collected data set.

[0041] S103, a region selection neural network selects multiple key regions of the hand bone image to obtain multiple key region images, wherein the region selection neural network is an initial region selection neural network trained using the collected data set.

[0042] S104, the main neural network extracts features of the multiple key area images, obtains multiple key area features of the hand bone images, and generates multiple key area feature vectors of the hand bone images. The main neural network performs key area diagnosis on the multiple key area feature vectors to obtain multiple key area diagnosis results.

[0043] S105, the main neural network splices the global feature vector and the key area feature vector to obtain a splicing vector, and the main neural network performs splicing diagnosis on the splicing vector to obtain a splicing diagnosis result.

[0044] S106, the main neural network performs a comprehensive diagnosis on the hand bone image according to the global diagnosis result, the key area diagnosis result and the splicing diagnosis result to obtain a final diagnosis result of whether the patient suffers from KBD.

[0045] From the above, it can be seen that the present application provides a method for analyzing hand bone images based on feature extraction. First, the trained main neural network extracts the global features of the patient's hand bone images to obtain a global feature vector, and then extracts the key area features by selecting the key area to obtain the key area feature vector, and then splices the global feature vector and the key area feature vector to obtain a splicing vector; the trained main neural network diagnoses the global feature vector, the key area feature vector and the splicing vector respectively to obtain a global diagnosis result, a key area diagnosis result and a splicing diagnosis result, and then performs a comprehensive diagnosis on the hand bone image based on the above three diagnosis results to determine whether the patient suffers from Kashin-Beck disease. The method provided in the present application ensures the efficiency and accuracy of the diagnosis of Kashin-Beck disease, meets the needs of large-scale screening of patients with Kashin-Beck disease, and solves the problem of low efficiency in manually judging whether an image is sick during large-scale screening in related technologies.

[0046] In some embodiments, the preprocessing includes segmenting and rotating the initial hand bone image, removing the background and other hand tissues, and obtaining a hand bone image with uniform size and maintained in a positive direction.

[0047] In a specific embodiment, according to the grayscale characteristics of the image, the bone and background segmentation can be completed by using the grayscale difference between the bone part and the background part. It can be obtained from the grayscale histogram of the initial hand bone image that a peak appears at a lower grayscale, and the pixels near this peak are marked as background and other tissues, while at a higher grayscale, the histogram is relatively flat, and the pixels here are the hand bone part. According to this logic, the tissue segmentation is completed.

[0048] In some embodiments, the method further comprises:

[0049] S107, constructing the initial main neural network and the initial area selection neural network.

[0050] S108, obtaining the data set, wherein the data set includes a plurality of initial sample hand bone images, diagnosis results of manually performing comprehensive diagnosis on the plurality of initial sample hand bone images, and coordinates of key areas manually selected on the plurality of initial sample hand bone images.

[0051] S109, the initial main neural network is trained based on the data set to obtain the main neural network.

[0052] S110, the initial region selection neural network is trained based on the data set to obtain the region selection neural network.

[0053] The initial sample hand bone images in the above-mentioned data set of this application are mainly X-ray images. The data acquisition is relatively convenient and the cost is low. At the same time, the initial main neural network and the initial area selection neural network are trained through the diagnosis results and the coordinates of the key areas, which ensures the reliability of the model at the data level.

[0054] In a specific embodiment, the data set includes X-ray images of adolescent hands collected in areas where Kaschin-Beck disease is prevalent; professional physicians manually label the images as whether they are diseased based on the clinical diagnostic criteria for Kaschin-Beck disease, and use the labels as labels for network learning; and professional physicians manually locate common lesion occurrence areas in some images and record the midpoint coordinates of these areas.

[0055] The initial region selection neural network is trained to extract the midpoint coordinates of the key region to ensure that the selection is more accurate. At the same time, it can select key regions for different selection ranges according to different regions, and then perform feature extraction.

[0056] In one embodiment, the initial main neural network is constructed using a ResNet network, and the initial region selection neural network is constructed using a ResNeXt network.

[0057] In a specific embodiment, the data set is divided into two sets, namely a training set and a test set. The test set data does not participate in the training process of the neural network, and data augmentation is performed only in the training set. After the neural network is trained, the best performing network model is saved to classify images on the test set, and the classification results are compared with the labels. The number of correct classifications in each case is counted to calculate common parameters for measuring the model, such as accuracy, precision, and recall, and the F1 index is used to neutralize the above indicators to intuitively evaluate the model performance.

[0058] In some embodiments, S109 includes:

[0059] S901, obtaining the initial sample hand bone image from the data set, and preprocessing the initial sample hand bone image to obtain a sample hand bone image.

[0060] S902, the initial main neural network performs global diagnosis processing on the sample hand bone image to obtain a sample global diagnosis result, and uses the sample global diagnosis result to calculate a global loss L g .

[0061] S903, the initial main neural network performs key area diagnosis processing on the sample hand bone image to obtain multiple sample key area diagnosis results, and calculates the key area loss L using the multiple sample key area diagnosis results p .

[0062] S904, the initial main neural network performs splicing diagnosis processing on the sample hand bone images to obtain multiple sample splicing diagnosis results, and calculates the splicing loss L using the multiple sample splicing diagnosis results * .

[0063] S905, the initial main neural network uses the global loss L g , the critical area loss L p And the splicing loss L * Calculate the overall network loss L total .

[0064] S906, based on the overall network loss L total The parameters of the initial main neural network are adjusted to complete a training of the initial main neural network.

[0065] S907, in response to determining the overall network loss L total All the data sets that are less than or equal to a preset threshold or the initial main neural network complete training, and the initial main neural network after training is used as the main neural network.

[0066] This application uses the transfer learning method to perform effective training on the data set, ensuring that the initial neural network can perform reliable feature extraction when the number of samples is relatively small.

[0067] In a specific embodiment, the initial sample hand bone image is preprocessed to obtain a sample hand bone image, which further includes: the initial sample hand bone image is obtained by splicing the hand bone images of four examinees, and four independent sample hand bone images with different directions are obtained after cross segmentation, and then straight line detection is used to find the straight line formed by the phalanges, and the direction of the straight line is taken as the direction of the central axis of the hand bone, and correction is performed in the vertical direction of the image to ensure that an upright sample hand bone image is obtained. Finally, the obtained image is size normalized to obtain a sample hand bone image that is ultimately used for training in the neural network. At the same time, considering the large difference in the number of positive and negative samples in the sample, appropriate image augmentation and balancing are performed on the sample hand bone image to improve the model expression ability.

[0068] In a specific embodiment, the global loss is used for feedback to improve the ability of the neural network to extract global features, and the key area loss is used to improve the ability of the neural network to obtain features of the lesion area. The initial main neural network uses the gradient descent algorithm with momentum as the optimization algorithm to minimize the above loss function, and iteratively updates the parameters in the neural network through the error back propagation method until the error is less than the preset threshold, or after reaching the preset number of iterations, the best performing set of parameters in the training process is saved as the main neural network. In this process, the global loss and the key area loss errors are also minimized, and the error back propagation method is used to adjust the parameters of the corresponding key area feature extraction network and the global feature extraction network to improve the feature extraction ability of the corresponding network.

[0069] In a specific embodiment, in the specific calculation of the loss function, a cross entropy loss function is used to calculate the error between the neural network diagnosis result and the data set. The calculation formula of the cross entropy function is:

[0070]

[0071] Where i represents y i Indicates the number of classifications of the sample. In one embodiment, i is 2, the positive sample label is 0, the negative sample label is 1, and p i Counted as the probability that the sample is predicted to be a positive example.

[0072] In some embodiments, the method further comprises:

[0073] S902 includes: the initial main neural network extracts and obtains the sample global features of the sample hand bone image, and generates a sample global feature vector of the sample hand bone image, and the initial main neural network performs a global diagnosis on the sample global feature vector to obtain the sample global diagnosis result.

[0074] S903 includes: the initial area selection neural network selects multiple key areas of the sample hand bone image to obtain multiple sample key area images; the initial main neural network extracts features of the multiple sample key area images to obtain multiple sample key area features of the sample hand bone images, and generates multiple sample key area feature vectors of the sample hand bone images; the initial main neural network performs key area diagnosis on the multiple sample key area feature vectors to obtain multiple sample key area diagnosis results.

[0075] S904 includes: the initial main neural network splices the sample global feature vector and the sample key area feature vector to obtain a sample splicing vector, and the initial main neural network performs splicing diagnosis on the sample splicing vector to obtain the sample splicing diagnosis result.

[0076] In some embodiments, S905 includes: the initial main neural network calculates the overall network loss L using the following formula: total :

[0077] L total =L * +λL g +βL p

[0078] The parameters λ and β are the global loss L g With the critical area loss L p The weight of .

[0079] In a specific embodiment, the parameters λ and β are preset to 1.

[0080] In some embodiments, S106 includes:

[0081] S601, the main neural network assigns different weights to the global diagnosis result, the key area diagnosis result and the splicing diagnosis result, and obtains the result value through calculation.

[0082] S602, the main neural network obtains the final diagnosis result by comparing the result value with a preset diagnosis threshold.

[0083] The main neural network uses features of multiple scales to make a decision simultaneously, and combines the three diagnosis results to obtain a final diagnosis result, thereby ensuring the accuracy of the final diagnosis result.

[0084] In some embodiments, the global features include the distribution of hand bones in the hand bone image and the texture of the bones; the key areas include the epiphysis of the middle phalanx and base phalanx of the ring finger, middle finger, and index finger, the epiphysis of the distal phalanx of the thumb, and the proximal end of the third metacarpal bone; the key area features include depression, sclerosis, and unevenness of the epiphysis.

[0085] A specific embodiment A is provided below for the hand bone image analysis method based on feature extraction of the present application.

[0086] Step 1: Data Preprocessing

[0087] The hand X-ray images used as the initial training set in the invention were collected by the team from the KBD epidemic area in Tibet. The annotation of the original sample images was completed by doctors to provide data for the subsequent network training. At the same time, each sample image was spliced ​​from the hand X-ray images of four subjects. In addition, in order to ensure that the network can focus on the hand bone structure, the sample images were preprocessed to obtain independent hand X-ray images including only the metacarpal and phalangeal bone structures. The data preprocessing process is as follows: Figure 2 shown.

[0088] Step 1.1: Initial sample image annotation

[0089] The labeling of the initial sample images of the hand used in the present invention is performed by professional physicians in accordance with the current domestic diagnostic standards for Kaschin-Beck disease, and the labeling is used as a label for network learning. At the same time, it is stipulated that if Kaschin-Beck disease is positive, that is, the image label of the patient is recorded as 1, otherwise the label is recorded as 0.

[0090] After the initial sample images are classified and labeled as diseased or not, some of the initial sample images are randomly divided as training samples. Common lesion sites are manually selected for these initial sample images, and the center coordinates of these areas are recorded to train the regional selection neural network for the subsequent automatic feature extraction of key areas.

[0091] Step 1.2: Initial sample image preprocessing

[0092] In order to remove the background of the image and other tissues outside the hand bones, the initial sample image needs to be segmented. The present invention starts from the different grayscale properties of different tissue structures in the X-ray image. According to the grayscale characteristics of the X-ray image, the segmentation of the bone and the background can be completed with the help of the grayscale difference between the bone part and the background part. It can be obtained from the grayscale histogram of the initial sample image that a peak appears at a lower grayscale, and the pixels near this place are marked as background and other tissues, while at a higher grayscale, the histogram is relatively flat, and the pixels here are the hand bone part. According to this logic, the segmentation of the tissue can be completed.

[0093] The initial sample image was obtained by stitching together the hand X-ray images of four examinees. After cross segmentation, four independent hand images with different directions were obtained. Then, straight line detection was used to find the straight line formed by the phalanges. The direction of the straight line was taken as the central axis of the hand, and correction was performed according to the vertical direction of the picture to ensure that an upright sample image was obtained.

[0094] Finally, the obtained sample images are resized and the sample image data used for training in the network is obtained. At the same time, considering the large difference in the number of positive and negative samples in the sample, appropriate image augmentation is performed on the sample image, and the sample is appropriately balanced to improve the model's expression ability.

[0095] Step 2: Image global feature extraction

[0096] The present invention uses the main neural network to extract the global features of the image, selects ResNet as the backbone network to build the overall model, and extracts the global features of the overall image and the features of the key areas in two parts. The operation flow chart of extracting the global features of hand X-ray images is shown in the figure. Figure 3 shown.

[0097] The neural network for extracting global features is composed of structures such as convolutional layers, pooling layers, and fully connected layers. The network uses convolutional layers to extract features and feature maps, and then the pooling layers compress the features and feature maps, extracting the main features while simplifying the network complexity. Finally, the fully connected layer sends the output to the classifier of the global feature extraction network for a global diagnosis.

[0098] The network input of the feature extraction part is the preprocessed hand bone X-ray image of uniform size obtained in step 1, and the output is the global feature vector of the image. The size of the global feature vector is T×1 this time.

[0099] In this section, the network that uses the global feature vector for classification is a single-layer fully connected neural network. The activation function is selected as the ReLU function. The fully connected layer transposes the weight matrix of the fully connected layer and multiplies it with the input vector. The output is obtained after adding the bias term. At this time, the fully connected layer completes the mapping of the global feature vector to the sample label space, that is, the category information of the image is obtained. While measuring the ability of the convolutional neural network to extract global features, this part of the classification results will also be used to calculate the network loss function in the future to further optimize the overall network performance.

[0100] Step 3: Automatically select common lesion areas and extract features

[0101] X-ray images of patients with certain orthopedic diseases often have significant lesions in certain locations. For example, in the early stage of osteomalacia, a notable manifestation on the X-ray images of the hands is the depression and sclerosis of the epiphysis of the digits. Due to the size of the feature area and the sample resolution, this information cannot be fully utilized in the feature extraction at the scale of the entire image. Therefore, it is necessary to select the key areas in the X-ray image and further extract the key area features through the convolutional neural network to assist in diagnosis. At the same time, as mentioned in step 2, the extracted key area features will also pass through the classification network to obtain the key area diagnosis results. This part of the results will be compared with the known labels, the loss function will be calculated, and the feedback will be given to the corresponding feature extraction network to correct the parameters. In addition, it will also be used to calculate the overall network loss function. The overall process is as follows: Figure 4 .

[0102] Step 3.1: Select the common lesion area

[0103] In the related technology, the key area is selected by selecting a fixed area based on experience to extract the features of the key area. However, due to the fact that the palm posture and palm size in different images are not fixed, the original method may not be able to correctly select the key area, resulting in failure to obtain the key area features, affecting the final diagnosis result.

[0104] According to the current diagnostic criteria for osteomalacia, the common locations of lesions examined are key locations including the epiphysis of the middle and coxa of the ring finger, middle finger, and index finger, the epiphysis of the distal truncus of the thumb, and the proximal end of the third metacarpal bone. Therefore, this application first manually annotates the midpoint coordinates of the above-mentioned areas in some data, saves the coordinate data as a training set, and trains the initial area selection neural network in the hope that it can locate the key points. The convolutional neural network used in this part is the ResNeXt network. After the above-mentioned key points are arranged in sequence, different selection ranges are selected according to their corresponding positions, and network training is started. During the training process, the obtained model is evaluated, and the size of the selection area is optimized until a more accurate model is obtained.

[0105] Using the trained model, the key areas of the sample image can be accurately located, key point annotation can be completed, and the key areas can be framed according to the optimized area size obtained previously to obtain a size-normalized key area image for subsequent feature extraction.

[0106] Step 3.2: Lesion area feature extraction

[0107] The features of the R selected key areas are extracted. Here, the value of R is 8. The feature extraction network is similar to the convolutional neural network structure used to extract global features in step 2. This part of the network is also composed of structures such as convolutional layers, pooling layers, and fully connected layers. Before feature extraction, the range of the selected area will be checked. If the selected area is out of bounds, feature extraction will not be performed. The network uses convolutional layers to extract feature vectors and feature maps from the R areas respectively, and uses pooling layers to compress them to obtain feature vectors of R lesion areas. The size of each vector is recorded as T×1. The obtained feature vector is input into the classification network with the structure shown in step 2. The fully connected layer is used to map the feature vector to the label space to obtain the predicted label, which is used to calculate the loss function later, feedback to adjust the network parameters, and improve the network's feature extraction effect.

[0108] Step 3.3: Vector stitching

[0109] The obtained global feature vector is concatenated with the key area feature vector to obtain a one-dimensional concatenated vector. The obtained one-dimensional vector inherits the global features from the entire image, and also integrates the features from the key area, supplementing the information on a smaller scale, so that the concatenated features can provide feedback from both the overall and local levels of the image, taking into account both integrity and specificity. Using the concatenated vector for classification, and combining the label calculation loss, correcting the network parameters can further improve the overall performance of the network.

[0110] The network for classifying concatenated vectors also uses a single-layer fully connected neural network to map the concatenated feature vector of size (T+T×R)×1 to the sample label space to complete the classification task. The classification result can more effectively reflect the feature extraction ability of the network, and also participate in the calculation of the loss function, feed back the neural network, and adjust the parameters. The process of vector concatenation is as follows: Figure 5 shown.

[0111] Step 4: Neural Network Training

[0112] The training process of the initial main neural network model in the present invention is as follows: Figure 6 shown.

[0113] In the process of training the neural network, considering the problem of data set size, the present invention uses the method of transfer learning to pre-train the initial main neural network on a large-scale public data set, and pre-set the network parameters based on this, which can significantly reduce the training cost and improve the overall model performance. For other networks in the overall model, the network parameters are set by random initialization. In addition, considering the size of the data set used in the present invention, a batch normalization module is added to the network structure to accelerate the convergence speed of the entire network, while reducing the model's demand for initialization, so that the network can accelerate convergence at a larger learning rate, and further improve the overall generalization performance of the network, and avoid overfitting caused by the size of the data set as much as possible.

[0114] For the calculation of the network loss function, the present invention defines the loss function as:

[0115] L total =L * +λL g +βL p

[0116] Among them, L total is the error between the overall network feature recognition and the label, L * is the diagnosis loss in the concatenated feature network, L g is the global loss, which is used for feedback to improve the network's ability to extract global features. p is the key area loss, which is used to improve the network's ability to obtain the characteristics of the lesion area. The parameters λ and β are used to adjust the weights of the global loss and the key area loss in the loss function calculation. They are preset to 1 during the model training process. The overall model uses the gradient descent algorithm with momentum as the optimization algorithm to minimize the above loss function and iteratively update the network parameters through the error back propagation method until the error is less than the preset threshold or reaches the preset number of iterations. The best performing set of parameters in the training process is saved as the network model, waiting for performance verification on the test set. In this process, Lp , L g The two errors are minimized, and the error back propagation method is used to adjust the parameters of the corresponding key area feature extraction network and the global feature extraction network to improve the feature extraction ability of the corresponding network.

[0117] During the network training process, the final diagnosis result of whether the patient is sick or not is given by majority judgment. Different weights are assigned to the diagnosis results of the global feature vector, key area feature vector and splicing feature vector, and the judgment thresholds of the three results are specified. The final score is calculated and compared with the threshold to complete the diagnosis of whether the patient is sick or not.

[0118] In the specific calculation of the network loss function, the present invention uses the cross entropy loss function to calculate the error between the network diagnosis result and the label. The calculation formula of the cross entropy function is:

[0119]

[0120] Where i represents y i Indicates the number of classifications of the sample, where i is 2, the positive sample label is 0, the negative sample label is 1, and p i Counted as the probability that the sample is predicted to be a positive example.

[0121] Step 5: Network performance verification

[0122] All data sets are divided into two sets, namely training set and test set. The data of both sets are labeled, but the test set data does not participate in the network training process, and data augmentation is performed only in the training set. Use the best network model saved after training in step 5 to classify images on the test set, compare the classification results with the labels, and count the number of correct classifications in each case to calculate common parameters for measuring the model, such as accuracy, precision, and recall. The F1 index is used to neutralize the above indicators to intuitively evaluate the model performance.

[0123] It should be noted that the method of the embodiment of the present application can be performed by a single device, such as a computer or server. The method of this embodiment can also be applied to a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of the embodiment of the present application, and the multiple devices will interact with each other to complete the described method.

[0124] It should be noted that the above describes some embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0125] Based on the same inventive concept, the present application also provides a hand bone image analysis device based on feature extraction, such as Figure 7 As shown, including:

[0126] A preprocessing module 10 is configured to preprocess the initial hand bone image of the patient to obtain a hand bone image;

[0127] The global diagnosis module 20 is configured to extract and obtain the global features of the hand bone image for the main neural network, and generate a global feature vector of the hand bone image, and the main neural network performs a global diagnosis on the global feature vector to obtain a global diagnosis result;

[0128] The key area selection module 30 is configured to select a plurality of key areas of the hand bone image using a region selection neural network to obtain a plurality of key area images;

[0129] The key area diagnosis module 40 is configured such that the main neural network extracts features of the plurality of key area images, obtains key area features of the plurality of hand bone images, and generates key area feature vectors of the plurality of hand bone images, and the main neural network performs key area diagnosis on the plurality of key area feature vectors to obtain a plurality of key area diagnosis results;

[0130] The splicing diagnosis module 50 is configured such that the main neural network splices the global feature vector and the key area feature vector to obtain a splicing vector, and the main neural network performs splicing diagnosis on the splicing vector to obtain a splicing diagnosis result;

[0131] The final result diagnosis module 60 is configured as the main neural network to perform a comprehensive diagnosis on the hand bone image according to the global diagnosis result, the key area diagnosis result and the splicing diagnosis result to obtain a final diagnosis result of whether the patient suffers from KBD.

[0132] The present application provides a hand bone image analysis device based on feature extraction, including a preprocessing module, a global diagnosis module, a key area selection module, a key area diagnosis module, a splicing diagnosis module and a final result diagnosis module. The present application provides a device, first, a trained main neural network extracts the global features of the patient's hand bone image to obtain a global feature vector, then extracts the key area features by selecting the key area to obtain the key area feature vector, and then splices the global feature vector and the key area feature vector to obtain a splicing vector; the trained main neural network diagnoses the global feature vector, the key area feature vector and the splicing vector respectively to obtain a global diagnosis result, a key area diagnosis result and a splicing diagnosis result, and then performs a comprehensive diagnosis on the hand bone image based on the above three diagnosis results to determine whether the patient suffers from Kashin-Beck disease. The method provided by the present application ensures the efficiency and accuracy of Kashin-Beck disease diagnosis, meets the needs of large-scale screening of patients with Kashin-Beck disease, and solves the problem of low efficiency in manually judging whether an image is sick during large-scale screening in related technologies.

[0133] In some embodiments, the apparatus further comprises:

[0134] A construction module is configured to construct the initial main neural network and the initial area selection neural network.

[0135] The acquisition module is configured to acquire the data set, which includes multiple initial sample hand bone images, diagnosis results of manually performing comprehensive diagnosis on the multiple initial sample hand bone images, and coordinates of key areas manually selected from the multiple initial sample hand bone images.

[0136] The main neural network training module is configured to train the initial main neural network based on the data set to obtain the main neural network.

[0137] The region box selection neural network training module is configured to train the initial region box selection neural network based on the data set to obtain the region box selection neural network.

[0138] In some embodiments, the main neural network training module includes:

[0139] The preprocessing submodule is configured to obtain the initial sample hand bone image from the data set, and preprocess the initial sample hand bone image to obtain a sample hand bone image.

[0140] The global diagnosis submodule is configured as the initial main neural network to perform global diagnosis processing on the sample hand bone image to obtain a sample global diagnosis result, and use the sample global diagnosis result to calculate the global loss L g .

[0141] The key area diagnosis submodule is configured as the initial main neural network to perform key area diagnosis processing on the sample hand bone image to obtain multiple sample key area diagnosis results, and calculate the key area loss L using the multiple sample key area diagnosis results. p .

[0142] The splicing diagnosis submodule is configured to perform splicing diagnosis processing on the sample hand bone images by the initial main neural network to obtain multiple sample splicing diagnosis results, and calculate the splicing loss L using the multiple sample splicing diagnosis results. * .

[0143] The first computing submodule is configured to use the global loss L g , the critical area loss L p And the splicing loss L * Calculate the overall network loss L total .

[0144] The training submodule is configured to be based on the overall network loss L total The parameters of the initial main neural network are adjusted to complete a training of the initial main neural network.

[0145] A response submodule is configured to respond to determining the overall network loss L total All the data sets that are less than or equal to a preset threshold or the initial main neural network complete training, and the initial main neural network after training is used as the main neural network.

[0146] In some embodiments, the apparatus further comprises:

[0147] The global diagnosis submodule is also configured as follows: the initial main neural network extracts and obtains the sample global features of the sample hand bone image, and generates a sample global feature vector of the sample hand bone image; the initial main neural network performs a global diagnosis on the sample global feature vector to obtain the sample global diagnosis result.

[0148] The key area diagnosis submodule is also configured as follows: the initial area selection neural network selects multiple key areas of the sample hand bone image to obtain multiple sample key area images; the initial main neural network extracts features of the multiple sample key area images to obtain multiple sample key area features of the sample hand bone images, and generates multiple sample key area feature vectors of the sample hand bone images; the initial main neural network performs key area diagnosis on the multiple sample key area feature vectors to obtain multiple sample key area diagnosis results.

[0149] The splicing diagnosis submodule is further configured as follows: the initial main neural network splices the sample global feature vector with the sample key area feature vector to obtain a sample splicing vector, and the initial main neural network performs splicing diagnosis on the sample splicing vector to obtain the sample splicing diagnosis result.

[0150] In some embodiments, the first calculation submodule includes: the initial main neural network calculates the overall network loss L using the following formula: total :

[0151] L total =L * +λL g +βL p

[0152] The parameters λ and β are the global loss L g With the critical area loss L p The weight of .

[0153] In some embodiments, the final result diagnosis module includes:

[0154] The second calculation submodule is configured such that the main neural network assigns different weights to the global diagnosis result, the key area diagnosis result and the splicing diagnosis result, and obtains the result value through calculation.

[0155] The final result diagnosis submodule is configured such that the main neural network obtains the final diagnosis result by comparing the result value with a preset diagnosis threshold.

[0156] In some embodiments, the global features include the distribution of hand bones in the hand bone image and the texture of the bones; the key areas include the epiphysis of the middle phalanx and base phalanx of the ring finger, middle finger, and index finger, the epiphysis of the distal phalanx of the thumb, and the proximal end of the third metacarpal bone; the key area features include depression, sclerosis, and unevenness of the epiphysis.

[0157] For the convenience of description, the above device is described in terms of functions divided into various modules. Of course, when implementing the present application, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0158] The device of the above embodiment is used to implement the corresponding hand bone image analysis method based on feature extraction in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0159] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, the present disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the hand bone image analysis method based on feature extraction described in any of the above embodiments is implemented.

[0160] Figure 8 A more specific schematic diagram of the hardware structure of an electronic device provided in this embodiment is shown, and the device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are connected to each other through the bus 1050 in the device.

[0161] The processor 1010 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0162] The memory 1020 may be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 may store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 1020 and are called and executed by the processor 1010.

[0163] The input / output interface 1030 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0164] The communication interface 1040 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0165] The bus 1050 includes a path that transmits information between the various components of the device (eg, the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040).

[0166] It should be noted that, although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040 and the bus 1050, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiments of the present specification, and does not necessarily include all the components shown in the figure.

[0167] The electronic device of the above embodiment is used to implement the corresponding hand bone image analysis method based on feature extraction in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0168] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the hand bone image analysis method based on feature extraction as described in any of the above embodiments.

[0169] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0170] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the hand bone image analysis method based on feature extraction as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0171] The embodiments corresponding to this application mainly achieve the following technical effects:

[0172] 1) The present application provides a method for analyzing hand bone images based on feature extraction. First, the trained main neural network extracts the global features of the patient's hand bone images to obtain a global feature vector, then extracts the key area features by selecting the key area to obtain a key area feature vector, and then splices the global feature vector and the key area feature vector to obtain a splicing vector; the trained main neural network diagnoses the global feature vector, the key area feature vector, and the splicing vector respectively to obtain a global diagnosis result, a key area diagnosis result, and a splicing diagnosis result, and then performs a comprehensive diagnosis on the hand bone image based on the above three diagnosis results to determine whether the patient suffers from Kashin-Beck disease. The method provided in the present application ensures the efficiency and accuracy of Kashin-Beck disease diagnosis, meets the needs of large-scale screening of patients with Kashin-Beck disease, and solves the problem of low efficiency in manually judging whether an image is sick during large-scale screening in related technologies.

[0173] 2) The present application provides a hand bone image analysis device based on feature extraction, including a preprocessing module, a global diagnosis module, a key area selection module, a key area diagnosis module, a splicing diagnosis module and a final result diagnosis module. The present application provides a device that firstly extracts the global features of the patient's hand bone image by a trained main neural network to obtain a global feature vector, then extracts the key area features by selecting the key area to obtain a key area feature vector, and then splices the global feature vector and the key area feature vector to obtain a splicing vector; the trained main neural network diagnoses the global feature vector, the key area feature vector and the splicing vector respectively to obtain a global diagnosis result, a key area diagnosis result and a splicing diagnosis result, and then performs a comprehensive diagnosis on the hand bone image based on the above three diagnosis results to determine whether the patient suffers from Kashin-Beck disease. The method provided by the present application ensures the efficiency and accuracy of Kashin-Beck disease diagnosis, meets the needs of large-scale screening of patients with Kashin-Beck disease, and solves the problem of low efficiency in manually judging whether an image is sick during large-scale screening in related technologies.

[0174] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present application (including the claims) is limited to these examples. In line with the concept of the present application, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.

[0175] In addition, to simplify the description and discussion, and in order not to make the embodiments of the present application difficult to understand, the known power supply / ground connection with the integrated circuit (IC) chip and other components may or may not be shown in the provided drawings. In addition, the device can be shown in the form of a block diagram to avoid making the embodiments of the present application difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform to be implemented in the embodiments of the present application (that is, these details should be fully within the scope of understanding of those skilled in the art). In the case of elaborating specific details (e.g., circuits) to describe exemplary embodiments of the present application, it is obvious to those skilled in the art that the embodiments of the present application can be implemented without these specific details or when these specific details are changed. Therefore, these descriptions should be considered to be illustrative rather than restrictive.

[0176] Although the present application has been described in conjunction with specific embodiments of the present application, many replacements, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0177] The embodiments of the present application are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the scope of protection of the present application.

Claims

1. A hand bone image analysis method based on feature extraction, characterized in that: include: The patient's initial hand bone image was preprocessed to obtain a hand bone image; The main neural network extracts and obtains the global features of the hand bone image, and generates a global feature vector of the hand bone image, and the main neural network performs a global diagnosis on the global feature vector to obtain a global diagnosis result, wherein the main neural network is obtained by training the initial main neural network using the collected data set; The region selection neural network selects multiple key regions of the hand bone image to obtain multiple key region images, wherein the region selection neural network is obtained by training the initial region selection neural network using the collected data set; The main neural network extracts features of the multiple key area images to obtain multiple key area features of the hand bone images, and generates multiple key area feature vectors of the hand bone images. The main neural network performs key area diagnosis on the multiple key area feature vectors to obtain multiple key area diagnosis results. The main neural network splices the global feature vector and the key area feature vector to obtain a splicing vector, and the main neural network performs splicing diagnosis on the splicing vector to obtain a splicing diagnosis result; The main neural network performs a comprehensive diagnosis on the hand bone image according to the global diagnosis result, the key area diagnosis result and the splicing diagnosis result to obtain a final diagnosis result of whether the patient suffers from KBD; Among them, it also includes: Constructing the initial main neural network and the initial region selection neural network; Acquire the data set, wherein the data set includes a plurality of initial sample hand bone images, diagnosis results of manually performing comprehensive diagnosis on the plurality of initial sample hand bone images, and coordinates of key areas manually selected on the plurality of initial sample hand bone images; Acquire the initial sample hand bone image from the data set, and preprocess the initial sample hand bone image to obtain a sample hand bone image; The initial main neural network performs global diagnosis processing on the sample hand bone image to obtain a sample global diagnosis result, and uses the sample global diagnosis result to calculate the global loss L g ; The initial main neural network performs key area diagnosis processing on the sample hand bone image to obtain multiple sample key area diagnosis results, and calculates the key area loss L using the multiple sample key area diagnosis results. p ; The initial main neural network performs splicing diagnosis processing on the sample hand bone images to obtain multiple sample splicing diagnosis results, and uses the multiple sample splicing diagnosis results to calculate the splicing loss L * ; The initial main neural network uses the global loss L g , the critical area loss L p And the splicing loss L * Calculate the overall network loss L total ; Based on the overall network loss L total Adjusting the parameters of the initial main neural network to complete a training of the initial main neural network; In response to determining the overall network loss L total All the data sets that are less than or equal to a preset threshold or the initial main neural network complete training, and the initial main neural network after training is used as the main neural network; The initial region selection neural network is trained based on the data set to obtain the region selection neural network.

2. The hand bone image analysis method based on feature extraction according to claim 1 is characterized in that: Also includes: The initial main neural network performs global diagnosis processing on the sample hand bone image to obtain a sample global diagnosis result, including: The initial main neural network extracts and obtains the sample global features of the sample hand bone image, and generates a sample global feature vector of the sample hand bone image, and the initial main neural network performs a global diagnosis on the sample global feature vector to obtain the sample global diagnosis result; The initial main neural network performs key area diagnosis processing on the sample hand bone image to obtain multiple sample key area diagnosis results, including: The initial region selection neural network selects multiple key regions of the sample hand bone image to obtain multiple sample key region images; The initial main neural network extracts features of the plurality of sample key area images, obtains sample key area features of the plurality of sample hand bone images, and generates sample key area feature vectors of the plurality of sample hand bone images, and the initial main neural network performs key area diagnosis on the plurality of sample key area feature vectors to obtain a plurality of sample key area diagnosis results; The initial main neural network performs splicing diagnosis processing on the sample hand bone images to obtain multiple sample splicing diagnosis results, including: The initial main neural network splices the sample global feature vector and the sample key area feature vector to obtain a sample splicing vector, and the initial main neural network performs splicing diagnosis on the sample splicing vector to obtain the sample splicing diagnosis result.

3. The hand bone image analysis method based on feature extraction according to claim 1, characterized in that: The initial main neural network uses the global loss L g , the critical area loss L p And the splicing loss L * Calculate the overall network loss L total ,include: The initial main neural network uses the following formula to calculate the overall network loss L total : L total =L * +λL g +βL p The parameters λ and β are the global loss L g With the critical area loss L p The weight of .

4. The hand bone image analysis method based on feature extraction according to claim 1, characterized in that: The main neural network performs a comprehensive diagnosis on the hand bone image according to the global diagnosis result, the key area diagnosis result and the spliced ​​diagnosis result to obtain a final diagnosis result of whether the patient suffers from KBD, including: The main neural network assigns different weights to the global diagnosis result, the key area diagnosis result and the splicing diagnosis result, and obtains the result value through calculation; The main neural network obtains the final diagnosis result by comparing the result value with a preset diagnosis threshold.

5. The hand bone image analysis method based on feature extraction according to claim 1, characterized in that: The global features include the distribution of hand bones in the hand bone image and the texture of the bones; the key areas include the epiphysis of the middle and base segments of the ring finger, middle finger, and index finger, the epiphysis of the distal segment of the thumb, and the proximal end of the third metacarpal bone; the key area features include depression, sclerosis, and unevenness of the epiphysis.

6. A hand bone image analysis device based on feature extraction, characterized in that: include: A preprocessing module is configured to preprocess the initial hand bone image of the patient to obtain a hand bone image; A global diagnosis module is configured to extract and obtain the global features of the hand bone image for the main neural network, and generate a global feature vector of the hand bone image, wherein the main neural network performs a global diagnosis on the global feature vector to obtain a global diagnosis result; wherein the main neural network is obtained by training the initial main neural network using the collected data set; A key area selection module is configured to select a plurality of key areas of the hand bone image using a region selection neural network to obtain a plurality of key area images; wherein the region selection neural network is obtained by training an initial region selection neural network using the collected data set; A key area diagnosis module is configured such that the main neural network extracts features of the plurality of key area images, obtains key area features of the plurality of hand bone images, and generates key area feature vectors of the plurality of hand bone images, and the main neural network performs key area diagnosis on the plurality of key area feature vectors to obtain a plurality of key area diagnosis results; A splicing diagnosis module is configured such that the main neural network splices the global feature vector with the key area feature vector to obtain a splicing vector, and the main neural network performs splicing diagnosis on the splicing vector to obtain a splicing diagnosis result; The final result diagnosis module is configured such that the main neural network performs a comprehensive diagnosis on the hand bone image according to the global diagnosis result, the key area diagnosis result and the spliced ​​diagnosis result to obtain a final diagnosis result of whether the patient suffers from KBD; The key area selection module is further configured to construct the initial main neural network and the initial area selection neural network; Acquire the data set, wherein the data set includes a plurality of initial sample hand bone images, diagnosis results of manually performing comprehensive diagnosis on the plurality of initial sample hand bone images, and coordinates of key areas manually selected on the plurality of initial sample hand bone images; Acquire the initial sample hand bone image from the data set, and preprocess the initial sample hand bone image to obtain a sample hand bone image; The initial main neural network performs global diagnosis processing on the sample hand bone image to obtain a sample global diagnosis result, and uses the sample global diagnosis result to calculate the global loss L g ; The initial main neural network performs key area diagnosis processing on the sample hand bone image to obtain multiple sample key area diagnosis results, and calculates the key area loss L using the multiple sample key area diagnosis results. p ; The initial main neural network performs splicing diagnosis processing on the sample hand bone images to obtain multiple sample splicing diagnosis results, and uses the multiple sample splicing diagnosis results to calculate the splicing loss L * ; The initial main neural network uses the global loss L g , the critical area loss L p And the splicing loss L * Calculate the overall network loss L total ; Based on the overall network loss L total Adjusting the parameters of the initial main neural network to complete a training of the initial main neural network; In response to determining the overall network loss L total All the data sets that are less than or equal to a preset threshold or the initial main neural network complete training, and the initial main neural network after training is used as the main neural network; The initial region selection neural network is trained based on the data set to obtain the region selection neural network.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 5 is implemented.

8. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to enable a computer to execute the method according to any one of claims 1 to 5.

Images