Bone age prediction method, device, electronic device and storage medium

Abstract

The present application provides a bone age prediction method, device, electronic device and storage medium, which belongs to the field of artificial intelligence technology. The method includes: obtaining a target bone image; performing key point detection on the target bone image through a first key point detection model to obtain a first target bone key point; performing a first posture correction process on the target bone image according to the first target bone key point to obtain a standard bone image; performing a segmentation process on the standard bone image through an image segmentation model to obtain a preliminary bone foreground map; performing key point detection on the standard bone image through a second key point detection model to obtain a second target bone key point; performing a second posture correction process on the preliminary bone foreground map according to the second target bone key point to obtain a standard bone foreground map; inputting the standard bone foreground map into the bone age prediction model to perform bone age prediction, and obtaining bone age prediction data. The present application can improve the accuracy of bone age prediction.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a bone age prediction method, device, electronic device and storage medium. Background Art

[0002] At present, most image-based bone age prediction methods need to rely on wrist bone development X-ray atlas (GP atlas method) or Tanner-Whitehouse bone age scoring method (TW method) to train neural network models, which often leads to poor robustness of the trained neural network model and affects the accuracy of bone age prediction. Therefore, how to improve the accuracy of bone age prediction has become a technical problem that needs to be solved urgently. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to propose a bone age prediction method, device, electronic device and storage medium, aiming to improve the accuracy of bone age prediction.

[0004] To achieve the above object, a first aspect of an embodiment of the present application proposes a bone age prediction method, the method comprising:

[0005] Acquire target bone image;

[0006] Performing key point detection on the target bone image using a pre-trained first key point detection model to obtain first target bone key points;

[0007] Performing a first posture correction process on the target skeleton image according to the first target skeleton key points to obtain a standard skeleton image;

[0008] Segmenting the standard bone image using a pre-trained image segmentation model to obtain a preliminary bone foreground image;

[0009] Performing key point detection on the standard bone image using a pre-trained second key point detection model to obtain second target bone key points;

[0010] Performing a second posture correction process on the preliminary skeleton foreground image according to the second target skeleton key points to obtain a standard skeleton foreground image;

[0011] The standard bone foreground image is input into a pre-trained bone age prediction model to perform bone age prediction and obtain bone age prediction data.

[0012] In some embodiments, the step of acquiring the target bone image includes:

[0013] Get the original bone image;

[0014] Scaling the original bone image according to preset size measurement parameters to obtain a first bone image;

[0015] Performing standardization processing on the first bone image to obtain a second bone image;

[0016] The second bone image is normalized to obtain the target bone image.

[0017] In some embodiments, the first key point detection model includes a first residual network, a first global convolutional network, and a first local convolutional network, and the step of performing key point detection on the target skeleton image using the pre-trained first key point detection model to obtain the first target skeleton key point includes:

[0018] Extracting features of the target bone image through the first residual network to obtain a target bone feature map;

[0019] Performing graph isomorphism processing on the target skeleton feature graph through the first global convolutional network to obtain a first initial skeleton key point;

[0020] The first initial skeleton key point is fine-tuned through the first local convolutional network to obtain the first target skeleton key point.

[0021] In some embodiments, the image segmentation model includes a convolutional layer, a first Transformer layer, and an image segmentation layer, and the step of performing segmentation processing on the standard bone image by using the pre-trained image segmentation model to obtain a preliminary bone foreground image includes:

[0022] Performing convolution processing on the standard bone image through a convolution layer to obtain standard bone feature data;

[0023] The standard bone feature data is encoded by the first Transformer layer to obtain a bone feature hidden vector;

[0024] The skeleton feature hidden vector is segmented by the image segmentation layer and preset pixel measurement parameters to obtain the preliminary skeleton foreground image.

[0025] In some embodiments, the second key point detection model includes a second residual network, a second global convolutional network, and a second local convolutional network, and the step of performing key point detection on the standard bone image using the pre-trained second key point detection model to obtain the second target bone key point includes:

[0026] Extracting features from the standard bone image using the second residual network to obtain a standard bone feature map;

[0027] Performing graph isomorphism processing on the standard skeleton feature graph through the second global convolutional network to obtain a second initial skeleton key point;

[0028] The second initial skeleton key point is fine-tuned through the second local convolutional network to obtain the second target skeleton key point.

[0029] In some embodiments, the bone age prediction model includes a third residual network, a second Transformer layer, and a fully connected layer, and the step of inputting the standard bone foreground image into a pre-trained bone age prediction model for bone age prediction to obtain bone age prediction data includes:

[0030] Inputting the standard bone foreground image into the bone age prediction model;

[0031] Performing feature extraction on the standard skeleton foreground image through the third residual network to obtain a plurality of standard skeleton foreground image slices;

[0032] encoding the standard skeleton foreground image slice through the second Transformer layer to obtain a standard skeleton foreground feature vector;

[0033] The bone age prediction data is obtained by performing bone age prediction on the standard bone foreground feature vector through the prediction function of the fully connected layer and the pre-acquired gender feature vector.

[0034] In some embodiments, before the step of inputting the standard bone foreground image into a pre-trained bone age prediction model to perform bone age prediction to obtain bone age prediction data, the method further includes:

[0035] Acquire the user gender data corresponding to the target skeleton image by means of a web crawler;

[0036] The user gender data is subjected to one-hot encoding processing to obtain the gender feature vector.

[0037] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a bone age prediction device, the device comprising:

[0038] An image acquisition module, used for acquiring a target bone image;

[0039] A first key point detection module, used to perform key point detection on the target bone image by using a pre-trained first key point detection model to obtain a first target bone key point;

[0040] A first image posture correction module, used for performing a first posture correction process on the target bone image according to the first target bone key points to obtain a standard bone image;

[0041] An image segmentation module is used to segment the standard bone image using a pre-trained image segmentation model to obtain a preliminary bone foreground image;

[0042] A second key point detection module, used to perform key point detection on the standard bone image by using a pre-trained second key point detection model to obtain a second target bone key point;

[0043] A second image posture correction module, used for performing a second posture correction process on the preliminary skeleton foreground image according to the second target skeleton key points to obtain a standard skeleton foreground image;

[0044] The bone age prediction module is used to input the standard bone foreground image into a pre-trained bone age prediction model to perform bone age prediction and obtain bone age prediction data.

[0045] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory, and the program, when executed by the processor, implements the method described in the first aspect above.

[0046] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a storage medium, which is a computer-readable storage medium used for computer-readable storage, and the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the method described in the first aspect above.

[0047] The bone age prediction method, device, electronic device and storage medium proposed in the present application obtain a target bone image, perform key point detection on the target bone image through a pre-trained first key point detection model, obtain a first target bone key point, and perform a first posture correction process on the target bone image according to the first target bone key point to obtain a standard bone image. In this way, the posture correction of the target bone image can be performed more accurately to obtain a standard bone image, thereby improving the image quality of the standard bone image. Furthermore, the standard bone image is segmented and processed by a pre-trained image segmentation model to obtain a preliminary bone foreground map. In this way, the standard bone image can be segmented and processed more accurately, and image data with low correlation with bone age prediction can be eliminated to obtain a preliminary bone foreground map for bone age prediction, thereby effectively reducing the amount of image data and improving prediction efficiency. Similarly, the standard bone image is subjected to key point detection through the pre-trained second key point detection model to obtain the second target bone key point, and the preliminary bone foreground image is subjected to the second posture correction processing according to the second target bone key point to obtain the standard bone foreground image. In this way, the preliminary bone foreground image can be more accurately posture corrected to obtain the standard bone foreground image, thereby improving the image quality of the standard bone foreground image. Finally, the standard bone foreground image is input into the pre-trained bone age prediction model for bone age prediction to obtain bone age prediction data, thereby improving the accuracy of bone age prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 is a flow chart of the bone age prediction method provided in an embodiment of the present application;

[0049] Figure 2 yes Figure 1 Flow chart of step S101 in FIG.

[0050] Figure 3 yes Figure 1 Flow chart of step S102 in FIG.

[0051] Figure 4 yes Figure 1 Flow chart of step S104 in FIG.

[0052] Figure 5 yes Figure 1 Flow chart of step S105 in FIG.

[0053] Figure 6 is another flow chart of the bone age prediction method provided in the embodiment of the present application;

[0054] Figure 7 yes Figure 1 Flow chart of step S107 in FIG.

[0055] Figure 8is a schematic diagram of the structure of a bone age prediction device provided in an embodiment of the present application;

[0056] Fig. 9 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0058] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0059] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0060] First, some nouns involved in this application are analyzed:

[0061] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0062] Natural language processing (NLP): NLP uses computers to process, understand and apply human languages ​​(such as Chinese, English, etc.). NLP is a branch of artificial intelligence and an interdisciplinary subject between computer science and linguistics. It is often referred to as computational linguistics. Natural language processing includes grammatical analysis, semantic analysis, and text understanding. Natural language processing is often used in technical fields such as machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information intent recognition, information extraction and filtering, text classification and clustering, public opinion analysis and opinion mining. It involves data mining related to language processing, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computing.

[0063] Information Extraction (NER): A text processing technology that extracts specified types of entity, relationship, event and other factual information from natural language text and forms structured data output. Information extraction is a technology that extracts specific information from text data. Text data is composed of some specific units, such as sentences, paragraphs, and chapters. Text information is composed of some small specific units, such as characters, words, phrases, sentences, paragraphs, or a combination of these specific units. Extracting noun phrases, names, place names, etc. from text data are all text information extraction. Of course, the information extracted by text information extraction technology can be various types of information.

[0064] Supervised learning: Supervised learning, also known as supervised training or learning with a teacher, refers to the process of adjusting the parameters of a classifier using a set of samples of known categories to achieve the required performance. Supervised learning is a machine learning task to infer a function from labeled training data. The training data includes a set of training instances, each of which consists of an input object (usually a vector) and a desired output value (also called a supervisory signal). The supervised learning algorithm analyzes the training data and produces an inferred function that can be used to map new instances.

[0065] Magnetic Resonance Imaging (MRI): MRI, also known as magnetic resonance imaging, is a physical phenomenon that is widely used in physics, chemistry, biology and other fields as an analytical method. MRI applies a certain frequency of radio frequency pulses to the human body in a static magnetic field, which excites the hydrogen protons in the human body and causes magnetic resonance.

[0066] Medical imaging: Medical imaging has multiple image modalities, such as magnetic resonance (MR), computed tomography (CT), PET, ultrasound (US) imaging, etc.

[0067] Residual Network: Residual network can fully utilize all hierarchical features of the original LR (Lightroom) image through residual dense block (RDB). For a very deep network, it is difficult and impractical to directly extract the output of each convolutional layer in the LR space. Use RDB as the building block of residual dense network (RDN). RDB contains densely connected layers and local feature fusion (LFF) with local residual learning (LRL). Each convolutional layer of RDB can access all subsequent layers and pass the information that needs to be retained. LFF extracts local dense features by adaptively preserving information by connecting the previous RDB with the state of all previous layers of the current RDB. In addition, LFF achieves extremely high growth rate by stabilizing the training of larger networks. After extracting multiple layers of local dense features, global feature fusion (GFF) is further performed to adaptively retain hierarchical features in a global manner. Each layer has direct access to the original LR input, resulting in implicit deep supervised learning.

[0068] Transformer layer: The transformer layer includes an attention layer, an add&norm layer, a feed forward layer, and an add&norm layer that are adjacent in sequence. In the attention layer, P input vectors are obtained from the previous layer of the transformer layer. With any first input vector among the P input vectors as the center, the intermediate vector corresponding to the first input vector is obtained based on the correlation between each input vector within the preset attention window range and the first input vector, so that the P intermediate vectors corresponding to the P input vectors are determined; in the pooling layer, the P intermediate vectors are merged into Q output vectors, wherein the multiple output vectors obtained by the transformer layer are used as feature representations of the current input.

[0069] At present, most image-based bone age prediction methods need to rely on wrist bone development X-ray atlas (GP atlas method) or Tanner-Whitehouse bone age scoring method (TW method) to train neural network models, which often leads to poor robustness of the trained neural network model and affects the accuracy of bone age prediction. Therefore, how to improve the accuracy of bone age prediction has become a technical problem that needs to be solved urgently.

[0070] Based on this, the embodiments of the present application provide a bone age prediction method, device, electronic device and storage medium, aiming to improve the accuracy of bone age prediction.

[0071] The bone age prediction method, device, electronic device and storage medium provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, the bone age prediction method in the embodiments of the present application is described.

[0072] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0073] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0074] The bone age prediction method provided in the embodiment of the present application relates to the field of artificial intelligence technology. The bone age prediction method provided in the embodiment of the present application can be applied to a terminal, can also be applied to a server side, and can also be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, and can also be configured to provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and cloud servers for basic cloud computing services such as big data and artificial intelligence platforms; the software can be an application for implementing the bone age prediction method, etc., but is not limited to the above forms.

[0075] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0076] Figure 1 is an optional flow chart of the bone age prediction method provided in the embodiment of the present application, Figure 1 The method may include but is not limited to steps S101 to S107.

[0077] Step S101, obtaining a target bone image;

[0078] Step S102, performing key point detection on the target skeleton image by using a pre-trained first key point detection model to obtain first target skeleton key points;

[0079] Step S103, performing a first posture correction process on the target skeleton image according to the first target skeleton key point to obtain a standard skeleton image;

[0080] Step S104, performing segmentation processing on the standard skeleton image by using a pre-trained image segmentation model to obtain a preliminary skeleton foreground image;

[0081] Step S105, performing key point detection on the standard skeleton image by using a pre-trained second key point detection model to obtain second target skeleton key points;

[0082] Step S106, performing a second posture correction process on the preliminary skeleton foreground image according to the second target skeleton key points to obtain a standard skeleton foreground image;

[0083] Step S107, inputting the standard bone foreground image into a pre-trained bone age prediction model to perform bone age prediction and obtain bone age prediction data.

[0084] In the step S101 to step S107 of the embodiment of the present application, the target bone image is subjected to key point detection by the first key point detection model trained in advance, and the first target bone key point is obtained. The target bone image is subjected to the first posture correction process according to the first target bone key point, and the standard bone image is obtained. The target bone image can be more accurately subjected to posture correction, and the standard bone image is obtained, thereby improving the image quality of the standard bone image. The standard bone image is subjected to segmentation process by the image segmentation model trained in advance, and a preliminary bone foreground map is obtained. The standard bone image can be more accurately subjected to segmentation process, and the image data with low correlation with bone age prediction is eliminated to obtain a preliminary bone foreground map for bone age prediction, thereby effectively reducing the image data volume and improving the prediction efficiency. Similarly, the standard bone image is subjected to key point detection by the second key point detection model trained in advance, and the second target bone key point is obtained. The preliminary bone foreground map is subjected to the second posture correction process according to the second target bone key point, thereby obtaining the standard bone foreground map. The preliminary bone foreground map can be more accurately subjected to posture correction, and the standard bone foreground map is obtained, thereby improving the image quality of the standard bone foreground map. Finally, the standard bone foreground image is input into the pre-trained bone age prediction model to perform bone age prediction and obtain bone age prediction data, which can improve the accuracy of bone age prediction.

[0085] See also Figure 2 In some embodiments, step S101 may include but is not limited to steps S201 to S204:

[0086] Step S201, obtaining an original bone image;

[0087] Step S202, scaling the original bone image according to preset size measurement parameters to obtain a first bone image;

[0088] Step S203, performing standardization processing on the first skeleton image to obtain a second skeleton image;

[0089] Step S204, normalizing the second skeleton image to obtain a target skeleton image.

[0090] In step S201 of some embodiments, an original bone image is acquired through Digital Imaging and Communications in Medicine (DICOM) or a medical database, wherein the original bone image is a three-dimensional image, which may be obtained through CT or MRI.

[0091] In some medical application scenarios, the above-mentioned original bone image can be a medical image, and the type of object contained in the original bone image is a lesion, that is, a part of the body where a disease occurs. Medical images refer to internal tissues obtained in a non-invasive manner for medical treatment or medical research, such as CT, MRI, ultrasound, X-ray images, and images generated by medical instruments using optical photography lights.

[0092] In step S202 of some embodiments, the original bone image can be scaled by using a resize function and preset size measurement parameters to change the width and height of the image to obtain a first bone image. For example, if the preset size measurement parameters require a width of 512 and a height of 512, the original bone image is scaled to a size of 512×512 by using a resize function.

[0093] In step S203 of some embodiments, the first bone image may be de-meaned to achieve centralization, so as to achieve standardization of the first bone image and obtain the second bone image. The standardization formula is shown in formula (1):

[0094]

[0095] Where x is the pixel matrix of the first bone image, μ is the mean of the reference bone image, and σ is the standard deviation of the reference bone image. Normalization can make the first bone image and the reference bone image have the same distribution (ie, the size and number of channels are the same).

[0096] In step S204 of some embodiments, the second bone image may be processed by a maximum-minimum normalization method to obtain a target bone image. The normalization formula is shown in formula (2):

[0097]

[0098] Among them, x i is the pixel value of the second bone image, max(x) is the maximum pixel value of the second bone image, and min(x) is the minimum pixel value of the second bone image.

[0099] Through the above steps S201 to S204, the original bone image can be processed, irrelevant information in the original bone image can be eliminated, useful real information can be restored, the detectability of relevant information can be enhanced, and the data can be simplified to the maximum extent to improve the reliability of bone age prediction.

[0100] See also Figure 3 In some embodiments, step S102 may include but is not limited to steps S301 to S303:

[0101] Step S301, extracting features of a target bone image through a first residual network to obtain a target bone feature map;

[0102] Step S302, performing graph isomorphism processing on the target skeleton feature graph through a first global convolutional network to obtain a first initial skeleton key point;

[0103] Step S303, fine-tuning the position of the first initial skeleton key point through the first local convolutional network to obtain the first target skeleton key point.

[0104] The first key point detection model includes a first residual network, a first global convolutional network and a first local convolutional network.

[0105] In step S301 of some embodiments, the first residual network includes a residual dense block formed by a series of 3*3 convolutional layers. The target bone image is directly output after convolution processing by the first residual dense block at the front end, and the second residual dense block at the back end can learn the difference between the input target bone image and the target bone image after processing by the first residual dense block at the front end, thereby outputting a target bone feature map.

[0106] In step S302 of some embodiments, image contour points are extracted from the target bone feature map through the graph isomorphism layer of the first global convolutional network to obtain the coordinates of the first initial key points that can characterize the contour of the target bone image, and the coordinates of the first initial key points are vectorized to obtain multiple 9-dimensional coordinate vectors, thereby obtaining the first initial target bone key points.

[0107] In step S303 of some embodiments, the error between each first initial target skeleton key point and the reference skeleton key point is calculated through the convolution layer of the first local convolutional network to obtain a first error value, and a first coordinate offset is generated based on the first error value. A two-dimensional vector corresponding to each first initial skeleton key point is generated based on the first coordinate offset at the output layer of the first local convolutional network, and the position of the first initial skeleton key point is fine-tuned based on the two-dimensional vector and the first coordinate offset to achieve the coordinate offset of the first initial skeleton key point and obtain the first target skeleton key point.

[0108] In step S103 of some embodiments, a first posture correction coefficient is generated according to the coordinate difference between the first target skeleton key point and the first reference key point, and the coordinates of each pixel in the target skeleton image are adjusted according to the first posture correction coefficient to obtain a standard skeleton image. The first posture correction coefficient can be determined according to the coordinate variance of the first target skeleton key point and the first reference key point, or can be calculated in other ways without limitation.

[0109] See also Figure 4In some embodiments, step S104 may include but is not limited to steps S401 to S403:

[0110] Step S401, performing convolution processing on the standard bone image through a convolution layer to obtain standard bone feature data;

[0111] Step S402, encoding the standard bone feature data through the first Transformer layer to obtain a bone feature hidden vector;

[0112] Step S403, segmenting the skeleton feature hidden vector through the image segmentation layer and the preset pixel metric parameters to obtain a preliminary skeleton foreground image.

[0113] Specifically, the image segmentation model is a TransUNet model, which includes a convolutional layer, a first Transformer layer and an image segmentation layer.

[0114] In step S401 of some embodiments, convolution processing is performed on the standard bone image by using the Transformer algorithm or bi-LSTM algorithm preset in the convolution layer to extract features of the standard bone image and obtain standard bone feature data.

[0115] In step S402 of some embodiments, the standard skeleton feature data is encoded using a Transformer algorithm or a bi-LSTM algorithm preset in the first Transformer layer to obtain a skeleton feature hidden vector.

[0116] In step S403 of some embodiments, the preset pixel metric parameters are mainly used to distinguish the foreground image and the background image of the standard skeleton image, and the specific values ​​of the pixel metric parameters can be set according to the actual situation without limitation. The pixel metric parameters are input into the decoder for image segmentation, and the decoder can perform main-background separation processing on the standard skeleton image, that is, the standard skeleton image is segmented into a preliminary skeleton foreground image and a preliminary skeleton background image.

[0117] See also Figure 5 In some embodiments, step S105 may also include but is not limited to steps S501 to S503:

[0118] Step S501, extracting features from the standard bone image through a second residual network to obtain a standard bone feature map;

[0119] Step S502, performing graph isomorphism processing on the standard skeleton feature graph through a second global convolutional network to obtain a second initial skeleton key point;

[0120] Step S503, fine-tuning the position of the second initial skeleton key point through the second local convolutional network to obtain the second target skeleton key point.

[0121] The second key point detection model includes a second residual network, a second global convolutional network, and a second local convolutional network.

[0122] In step S501 of some embodiments, the second residual network includes a residual dense block formed by at least two 3*3 convolutional layers connected in series. The standard bone image is directly output after convolution processing by the third residual dense block at the front end. The fourth residual dense block at the back end can learn the difference between the input standard bone image and the standard bone image after processing by the third residual dense block at the front end, thereby outputting a standard bone feature map.

[0123] In step S502 of some embodiments, image contour points are extracted from the standard bone feature map through the graph isomorphism layer of the second global convolutional network to obtain the coordinates of the second initial key points that can characterize the contour of the standard bone image, and the coordinates of the second initial key points are vectorized to obtain multiple 9-dimensional coordinate vectors, thereby obtaining the second initial target bone key points.

[0124] In step S503 of some embodiments, the error between each second initial target skeleton key point and the reference skeleton key point is calculated through the convolution layer of the second local convolutional network to obtain a second error value, and a second coordinate offset is generated based on the second error value. A two-dimensional vector corresponding to each second initial skeleton key point is generated based on the second coordinate offset at the output layer of the second local convolutional network, and the position of the second initial skeleton key point is fine-tuned based on the two-dimensional vector and the second coordinate offset to achieve the coordinate offset of the second initial skeleton key point and obtain the second target skeleton key point.

[0125] In step S106 of some embodiments, a second posture correction coefficient is generated according to the coordinate difference between the second target skeleton key point and the second reference key point, and the coordinates of each pixel point in the preliminary skeleton foreground image are adjusted according to the second posture correction coefficient to obtain a standard skeleton foreground image. The second posture correction coefficient can be determined according to the coordinate variance of the second target skeleton key point and the second reference key point, or can be calculated in other ways without limitation.

[0126] See also Figure 6 In some embodiments, before step S107, the method further includes but is not limited to steps S601 to S602:

[0127] Step S601, obtaining user gender data corresponding to the target skeleton image by means of a web crawler;

[0128] Step S602: Perform one-hot encoding on the user gender data to obtain a gender feature vector.

[0129] In step S601 of some embodiments, a web crawler is written and a data source is set to crawl data in a targeted manner to obtain target flow survey data. It should be noted that the target flow survey data includes the gender data of the user corresponding to the target eye image.

[0130] Furthermore, according to the preset category labels and sequence classifiers, the target epidemiological data is subjected to label classification processing to obtain annotated epidemiological data. Specifically, the entity features of the target epidemiological data are marked using pre-trained sequence classifiers and feature category labels, so that these entity features can all be labeled with preset labels to improve classification efficiency. Among them, the feature category labels include gender labels, name labels, etc. The pre-trained sequence classifier can be a model based on the conditional random field algorithm (CRF) or a model based on the bidirectional long short-term memory algorithm (bi-LSTM). For example, a sequence classifier can be constructed based on the bi-LSTM algorithm. In the model based on the bi-LSTM algorithm, the word wi and character embedding of the target epidemiological data are input, and a single output layer is generated at the position where the output is connected through the left-to-right long short-term memory and the right-to-left long short-term memory. The sequence classifier can pass the input entity features directly to the softmax classifier through this output layer. The softmax classifier creates a probability distribution on the preset feature category labels, so as to label and classify the entity features according to the probability distribution, and identify the user gender data corresponding to the gender label from the entity features with multiple different feature category labels. Furthermore, the convolution layer performs convolution processing on the labeled flow data to extract the user gender data corresponding to the corresponding gender label.

[0131] It should be noted that One-Hot encoding is also called one-bit effective encoding. The method is to use an N-bit state register to encode N states. Each state has its own independent register bit, and only one state is valid at any time. One-hot encoding is to use a vector of length V to represent the gender data of each user.

[0132] In step S602 of some embodiments, the user gender data can be represented as a vector through one-hot encoding, with male gender recorded as [0,1] and female gender recorded as [1,0], thereby obtaining a gender feature vector corresponding to each standard fundus image.

[0133] See also Figure 7 In some embodiments, step S107 may also include but is not limited to steps S701 to S704:

[0134] Step S701, inputting the standard bone foreground image into the bone age prediction model;

[0135] Step S702, extracting features of the standard skeleton foreground image through a third residual network to obtain a plurality of standard skeleton foreground image slices;

[0136] Step S703, encoding the standard skeleton foreground image slices through the second Transformer layer to obtain a standard skeleton foreground feature vector;

[0137] Step S704, predicting the bone age of the standard bone foreground feature vector using the prediction function of the fully connected layer and the pre-acquired gender feature vector to obtain bone age prediction data.

[0138] Specifically, the bone age prediction model is a Vision Transformer model, which includes a third residual network, a second Transformer layer, and a fully connected layer.

[0139] In step S701 and step S702 of some embodiments, the standard bone foreground image is input into the bone age prediction model. The standard bone foreground image is feature extracted by the third residual network of the bone age prediction model, wherein the third residual network includes at least two residual dense blocks formed by 3*3 convolution layers connected in series, and the standard bone foreground image is directly output after convolution processing by the fifth residual dense block at the front end, and the sixth residual dense block at the back end can learn the difference between the input standard bone foreground image and the standard bone foreground image after processing by the sixth residual dense block at the front end, thereby outputting the standard bone foreground feature map, and further, the standard bone foreground image is sliced ​​according to a preset slice size parameter or according to a preset slice distance to obtain a plurality of standard bone foreground image slices.

[0140] In step S703 of some embodiments, the standard skeleton foreground image slices are encoded using a Transformer algorithm or a bi-LSTM algorithm preset in the second Transformer layer to obtain a standard skeleton foreground feature vector.

[0141] In step S704 of some embodiments, first, the standard bone foreground feature vector is mapped to a preset vector space through the MLP network in the fully connected layer, so that the mapped standard bone foreground feature vector is in the preset feature dimension, that is, the standard bone foreground feature vector is converted into a 1000-dimensional vector through the MLP network, and at the same time, the gender feature vector is converted into a 320-dimensional vector through the MLP network. Further, the gender feature vector in 320 dimensions and the standard bone foreground feature vector in 1000 dimensions are spliced ​​to obtain a bone age prediction feature vector; the splicing of the gender feature vector and the standard bone foreground feature vector in this embodiment is to add the gender feature vector to the standard bone foreground feature vector. Finally, the bone age prediction feature vector is assigned to the prediction function, wherein the prediction function is a softmax function, and a probability distribution is created on the preset bone age interval category label through the softmax function, so that the bone age prediction feature vector is calculated according to the probability distribution, and the bone age probability value of each bone age interval category is obtained, and the bone age interval corresponding to the maximum bone age probability value is selected as the final bone age prediction interval to obtain bone age prediction data.

[0142] The embodiment of the present application obtains the target bone image, performs key point detection on the target bone image through the first key point detection model trained in advance, obtains the first target bone key point, and performs the first posture correction processing on the target bone image according to the first target bone key point to obtain the standard bone image, through this method, the target bone image can be more accurately posture corrected to obtain the standard bone image, and the image quality of the standard bone image is improved. Further, the standard bone image is segmented by the image segmentation model trained in advance to obtain a preliminary bone foreground map, so that the standard bone image can be segmented more accurately, and the image data with low correlation with bone age prediction is eliminated to obtain a preliminary bone foreground map for bone age prediction, which effectively reduces the image data volume and improves the prediction efficiency. Similarly, the standard bone image is key point detected by the second key point detection model trained in advance to obtain the second target bone key point, and the preliminary bone foreground map is subjected to the second posture correction processing according to the second target bone key point to obtain the standard bone foreground map, through this method, the preliminary bone foreground map can be more accurately posture corrected to obtain the standard bone foreground map, and the image quality of the standard bone foreground map is improved. Finally, the standard skeleton foreground image is input into the pre-trained bone age prediction model for bone age prediction, and bone age prediction data is obtained, which can improve the accuracy of bone age prediction. In addition, the embodiment of the present application can improve the image quality to a large extent by processing the image of the target bone image through the first key point detection model, the second key point detection model, and the image segmentation model, so that the standard skeleton foreground image input into the bone age prediction model can have a better prediction effect, and can also better improve the robustness of the entire bone age prediction process.

[0143] See also Figure 8 The present application also provides a bone age prediction device, which can implement the above-mentioned bone age prediction method, and the device includes:

[0144] An image acquisition module 801 is used to acquire a target bone image;

[0145] A first key point detection module 802 is used to perform key point detection on the target skeleton image by using a pre-trained first key point detection model to obtain a first target skeleton key point;

[0146] A first image posture correction module 803, used for performing a first posture correction process on the target skeleton image according to the first target skeleton key point to obtain a standard skeleton image;

[0147] An image segmentation module 804 is used to segment the standard bone image using a pre-trained image segmentation model to obtain a preliminary bone foreground image;

[0148] A second key point detection module 805 is used to perform key point detection on the standard bone image by using a pre-trained second key point detection model to obtain a second target bone key point;

[0149] A second image posture correction module 806 is used to perform a second posture correction process on the preliminary skeleton foreground image according to the second target skeleton key points to obtain a standard skeleton foreground image;

[0150] The bone age prediction module 807 is used to input the standard bone foreground image into the pre-trained bone age prediction model to perform bone age prediction and obtain bone age prediction data.

[0151] The specific implementation of the bone age prediction device is basically the same as the specific implementation of the above-mentioned bone age prediction method, and will not be repeated here.

[0152] The embodiment of the present application also provides an electronic device, the electronic device comprising: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory, and the above-mentioned bone age prediction method is realized when the program is executed by the processor. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0153] See also Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0154] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0155] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the bone age prediction method of the embodiment of this application;

[0156] Input / output interface 903, used to implement information input and output;

[0157] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0158] A bus 905 that transmits information between the various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0159] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0160] An embodiment of the present application also provides a storage medium, which is a computer-readable storage medium used for computer-readable storage. The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the above-mentioned bone age prediction method.

[0161] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0162] The bone age prediction method, bone age prediction device, electronic device and storage medium provided in the embodiment of the present application obtain a target bone image, perform key point detection on the target bone image through a pre-trained first key point detection model, obtain a first target bone key point, and perform a first posture correction process on the target bone image according to the first target bone key point to obtain a standard bone image. In this way, the posture correction of the target bone image can be performed more accurately to obtain a standard bone image, thereby improving the image quality of the standard bone image. Further, the standard bone image is segmented and processed by a pre-trained image segmentation model to obtain a preliminary bone foreground map. In this way, the standard bone image can be segmented and processed more accurately, and image data with low correlation with bone age prediction is eliminated to obtain a preliminary bone foreground map for bone age prediction, thereby effectively reducing the amount of image data and improving prediction efficiency. Similarly, key point detection is performed on the standard skeleton image by the second key point detection model trained in advance, and the second target skeleton key point is obtained, and the second posture correction process is performed to the preliminary skeleton foreground map according to the second target skeleton key point, and the standard skeleton foreground map is obtained, and the posture correction can be performed to the preliminary skeleton foreground map more accurately in this way, and the standard skeleton foreground map is obtained, and the image quality of the standard skeleton foreground map is improved. Finally, the standard skeleton foreground map is input to the pre-trained bone age prediction model for bone age prediction, and bone age prediction data is obtained, which can improve the accuracy of bone age prediction. In addition, the embodiment of the present application can improve the image quality to a large extent by the first key point detection model, the second key point detection model, and the image segmentation model for the image processing of the target skeleton image, so that the standard skeleton foreground map input to the bone age prediction model can have a better prediction effect, and the robustness of the whole bone age prediction process can also be improved well.

[0163] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0164] It can be understood by those skilled in the art that Figure 1-7 The technical solutions shown in the figure do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than those shown in the figure, or a combination of certain steps, or different steps.

[0165] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0166] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0167] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0168] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0169] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0170] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0171] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0172] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0173] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A bone age prediction method, characterized in that: The method comprises: Acquire target bone image; Performing key point detection on the target bone image using a pre-trained first key point detection model to obtain first target bone key points; Performing a first posture correction process on the target skeleton image according to the first target skeleton key points to obtain a standard skeleton image; Segmenting the standard bone image using a pre-trained image segmentation model to obtain a preliminary bone foreground image; Performing key point detection on the standard bone image using a pre-trained second key point detection model to obtain second target bone key points; Performing a second posture correction process on the preliminary skeleton foreground image according to the second target skeleton key points to obtain a standard skeleton foreground image; Inputting the standard bone foreground image into a pre-trained bone age prediction model to perform bone age prediction to obtain bone age prediction data; The bone age prediction model includes a third residual network, a second Transformer layer and a fully connected layer. The standard bone foreground image is input into the pre-trained bone age prediction model for bone age prediction to obtain bone age prediction data, including: Inputting the standard bone foreground image into the bone age prediction model; Performing feature extraction on the standard skeleton foreground image through the third residual network to obtain a plurality of standard skeleton foreground image slices; encoding the standard skeleton foreground image slice through the second Transformer layer to obtain a standard skeleton foreground feature vector; Performing bone age prediction on the standard bone foreground feature vector through the prediction function of the fully connected layer and the pre-acquired gender feature vector to obtain the bone age prediction data; The gender feature vector is obtained by obtaining the user gender data corresponding to the target skeleton image by means of a web crawler; and performing one-hot encoding processing on the user gender data.

2. The bone age prediction method according to claim 1, characterized in that: The step of obtaining the target bone image comprises: Get the original bone image; Scaling the original bone image according to preset size measurement parameters to obtain a first bone image; Performing standardization processing on the first bone image to obtain a second bone image; The second bone image is normalized to obtain the target bone image.

3. The bone age prediction method according to claim 1, characterized in that: The first key point detection model includes a first residual network, a first global convolutional network and a first local convolutional network. The step of performing key point detection on the target skeleton image through the pre-trained first key point detection model to obtain the first target skeleton key point includes: Extracting features of the target bone image through the first residual network to obtain a target bone feature map; Performing graph isomorphism processing on the target skeleton feature graph through the first global convolutional network to obtain a first initial skeleton key point; The first initial skeleton key point is fine-tuned through the first local convolutional network to obtain the first target skeleton key point.

4. The bone age prediction method according to claim 1, characterized in that: The image segmentation model includes a convolutional layer, a first Transformer layer and an image segmentation layer. The step of segmenting the standard bone image using the pre-trained image segmentation model to obtain a preliminary bone foreground image includes: Performing convolution processing on the standard bone image through a convolution layer to obtain standard bone feature data; The standard bone feature data is encoded by the first Transformer layer to obtain a bone feature hidden vector; The skeleton feature hidden vector is segmented by the image segmentation layer and preset pixel measurement parameters to obtain the preliminary skeleton foreground image, wherein the preset pixel measurement parameters are used to distinguish the foreground image and the background image of the standard skeleton image.

5. The bone age prediction method according to claim 1, characterized in that: The second key point detection model includes a second residual network, a second global convolutional network and a second local convolutional network. The step of performing key point detection on the standard bone image by using the pre-trained second key point detection model to obtain the second target bone key point includes: Extracting features from the standard bone image using the second residual network to obtain a standard bone feature map; Performing graph isomorphism processing on the standard skeleton feature graph through the second global convolutional network to obtain a second initial skeleton key point; The second initial skeleton key point is fine-tuned through the second local convolutional network to obtain the second target skeleton key point.

6. A bone age prediction device, characterized in that: The device comprises: An image acquisition module, used for acquiring a target bone image; A first key point detection module, used to perform key point detection on the target bone image by using a pre-trained first key point detection model to obtain a first target bone key point; A first image posture correction module, used for performing a first posture correction process on the target bone image according to the first target bone key points to obtain a standard bone image; An image segmentation module, used to perform segmentation processing on the standard bone image through a pre-trained image segmentation model to obtain a preliminary bone foreground image; A second key point detection module, used to perform key point detection on the standard bone image by using a pre-trained second key point detection model to obtain a second target bone key point; A second image posture correction module, used for performing a second posture correction process on the preliminary skeleton foreground image according to the second target skeleton key points to obtain a standard skeleton foreground image; A bone age prediction module is used to input the standard bone foreground image into a pre-trained bone age prediction model to perform bone age prediction and obtain bone age prediction data; The bone age prediction model includes a third residual network, a second Transformer layer and a fully connected layer. The standard bone foreground image is input into the pre-trained bone age prediction model for bone age prediction to obtain bone age prediction data, including: Inputting the standard bone foreground image into the bone age prediction model; Performing feature extraction on the standard skeleton foreground image through the third residual network to obtain a plurality of standard skeleton foreground image slices; encoding the standard skeleton foreground image slice through the second Transformer layer to obtain a standard skeleton foreground feature vector; Performing bone age prediction on the standard bone foreground feature vector through the prediction function of the fully connected layer and the pre-acquired gender feature vector to obtain the bone age prediction data; The gender feature vector is obtained by obtaining the user gender data corresponding to the target skeleton image by means of a web crawler; and performing one-hot encoding processing on the user gender data.

7. An electronic device, characterized in that: The electronic device includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory. When the program is executed by the processor, the steps of the bone age prediction method as described in any one of claims 1 to 5 are realized.

8. A storage medium, the storage medium being a computer-readable storage medium, used for computer-readable storage, characterized in that: The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the bone age prediction method described in any one of claims 1 to 5.

Images