Training method, identification method and device for elbow joint rotation axis identification model

Abstract

The present disclosure relates to the field of image processing technology, and provides a training method, an identification method and a device for an elbow rotation axis recognition model. The method includes: obtaining a three-dimensional joint image, the three-dimensional joint image carries at least two joint rotation axis annotations marked by annotators; projecting the three-dimensional joint image and the joint rotation axis annotations it carries onto a two-dimensional image imaging plane to obtain a simulated two-dimensional image; performing image modality conversion on the simulated two-dimensional image to obtain a converted two-dimensional image; using the converted two-dimensional image to train an initial joint rotation axis recognition model until a preset training termination condition is reached to obtain a final joint rotation axis recognition model. The final joint rotation axis recognition model disclosed in the present disclosure can accurately identify the rotation axis position of a joint image taken by a patient in any posture, greatly reducing the difficulty of filming operations, and only requires one shot to meet the requirements, with simple operation, short time consumption and low radiation dose.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to a training method, an identification method and a device for an elbow joint rotation axis identification model. Background Art

[0002] Human joints (such as elbows, knees, and knuckles) are roughly equivalent to hinge structures. During movement, the two bony structures related to the hinge rotate around a common point, which is the center of rotation. External fixation of the elbow is an important method for treating complex elbow instability, and accurate positioning of the central axis of elbow rotation is the core of external fixation of the elbow, and is also the key to improving the overall treatment effect and reducing complications.

[0003] In the prior art, when positioning the central axis of elbow rotation, the elbow joint needs to be strictly placed in a special position, namely, the standard elbow joint anteroposterior position. Among them, the standard elbow joint anteroposterior position includes the standard elbow joint lateral position and the standard elbow joint anterior position. In the standard elbow joint lateral position, the examinee sits in front of the photographic table, with the elbow bent to 90°, the ulnar side down, the shoulder lowered, the elbow close to the detector, and the center line is aligned with the lateral epicondyle of the humerus and vertically enters the detector. In the standard elbow joint anterior position, the examinee sits in front of the photographic table, with the palm facing up, the elbow and forearm fully extended, the shoulder lowered, the back of the elbow close to the detector, and the center line is aligned with the midpoint of the line connecting the internal and external condyles of the humerus and vertically enters the detector. However, since patients often face limited abduction and internal rotation of the shoulder joint, or fractures that have been repaired during surgery, and ligaments may move again during the placement of this position, it is often difficult to capture the patient's standard elbow joint anteroposterior position image during surgery. In addition, this method requires repeated X-ray fluoroscopy, and can only subjectively judge the central axis of rotation of the elbow joint based on the personal experience of medical staff. It is also necessary to insert an axis positioning needle after repeated X-ray fluoroscopy verification. The operation is cumbersome, the filming requirements are high, the time is long, and it is difficult to accurately identify the rotation axis position of the joint.

[0004] Therefore, the existing technology requires that the joint be strictly placed in the standard elbow joint anteroposterior position during intraoperative radiography, which makes the radiography operation difficult and requires multiple repeated radiography. The operation is cumbersome, time-consuming, and involves a large radiation dose. It is also difficult to accurately identify the position of the rotation axis of the joint, resulting in low accuracy and safety of the surgical results. Summary of the invention

[0005] In view of this, the embodiments of the present disclosure provide a training method, an identification method and a device for an elbow joint rotation axis recognition model to solve the problem that the prior art requires that the joint be strictly placed in the standard elbow joint anteroposterior position during surgery, the filming operation is difficult, and multiple repeated films are required. The operation is cumbersome, time-consuming, and the radiation dose is large. At the same time, it is also difficult to accurately identify the position of the joint's rotation axis, resulting in low accuracy and safety of the surgical results.

[0006] A first aspect of the embodiments of the present disclosure provides a training method for an elbow joint rotation axis recognition model, comprising:

[0007] Acquire a three-dimensional image of the joint, wherein the three-dimensional image of the joint carries a joint rotation axis annotation marked by an annotator;

[0008] Projecting the joint three-dimensional image and the joint rotation axis annotation carried by it onto a preset two-dimensional image imaging plane to obtain a simulated two-dimensional image;

[0009] Performing image modality conversion on the simulated two-dimensional image to obtain a converted two-dimensional image;

[0010] The initial joint rotation axis recognition model is trained using the converted two-dimensional image until a preset training termination condition is reached to obtain a final joint rotation axis recognition model.

[0011] A second aspect of the embodiments of the present disclosure provides a method for identifying an elbow joint rotation axis, comprising:

[0012] Acquire a perspective image of a joint to be identified;

[0013] The joint perspective image to be identified is input into the above-mentioned final joint rotation axis recognition model, and the rotation axis recognition result of the joint perspective image is output.

[0014] A third aspect of the embodiments of the present disclosure provides a training device for an elbow joint rotation axis recognition model, comprising:

[0015] An image acquisition module is configured to acquire a three-dimensional image of a joint, wherein the three-dimensional image of the joint carries a joint rotation axis annotation marked by an annotator;

[0016] An image simulation module is configured to project the joint three-dimensional image and the joint rotation axis annotation carried by it onto a preset two-dimensional image imaging plane to obtain a simulated two-dimensional image;

[0017] A modality conversion module is configured to perform image modality conversion on the simulated two-dimensional image to obtain a converted two-dimensional image;

[0018] The model training module is configured to use the converted two-dimensional image to train the initial joint rotation axis recognition model until a preset training termination condition is reached to obtain a final joint rotation axis recognition model.

[0019] According to a fourth aspect of the embodiments of the present disclosure, there is provided a device for identifying an elbow joint rotation axis, comprising:

[0020] An acquisition module, configured to acquire a perspective image of a joint to be identified;

[0021] The image recognition module is configured to input the joint perspective image to be recognized into the above-mentioned final joint rotation axis recognition model, and output the rotation axis recognition result of the joint perspective image.

[0022] According to a fifth aspect of an embodiment of the present disclosure, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0023] According to a sixth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, which stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0024] In a seventh aspect of the embodiments of the present disclosure, a robotic surgery system is provided, comprising a robotic arm, a host computer, a cone beam computed tomography scanner, an optical tracking system, a positioning ruler, and the above-mentioned elbow joint rotation axis identification device.

[0025] Compared with the prior art, the disclosed embodiment has at least the following beneficial effects: the disclosed embodiment acquires a three-dimensional joint image, and the three-dimensional joint image carries a joint rotation axis annotation marked by an annotator; the three-dimensional joint image and the joint rotation axis annotation carried by the annotator are projected together to a preset two-dimensional image imaging plane to obtain a simulated two-dimensional image; the simulated two-dimensional image is subjected to image modality conversion to obtain a converted two-dimensional image; the converted two-dimensional image is used to train an initial joint rotation axis recognition model until a preset training termination condition is reached to obtain a final joint rotation axis recognition model. The final joint rotation axis recognition model can accurately identify the rotation axis position of the joint image taken by the patient in any position, and there is no need to strictly require the patient to place the joint in the standard elbow joint lateral position during intraoperative filming, which greatly reduces the difficulty of filming operation, and only one shot is required to achieve the requirement of accurately identifying the joint rotation axis, simplifying the surgical operation, shortening the time consumption, reducing intraoperative X-ray exposure, that is, reducing the radiation dose, and improving the accuracy, reliability and safety of the surgical results. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0027] Figure 1 It is a flowchart of a training method for an elbow joint rotation axis recognition model provided by an embodiment of the present disclosure;

[0028] Figure 2 It is a schematic diagram of a simulated projection of a three-dimensional image of a joint in a training method for an elbow joint rotation axis recognition model provided by an embodiment of the present disclosure;

[0029] Figure 3 It is a flowchart of a method for identifying the rotation axis of an elbow joint provided by an embodiment of the present disclosure;

[0030] Figure 4 is a schematic diagram of the structure of a recurrent adversarial generative network provided by an embodiment of the present disclosure;

[0031] Figure 5 It is a structural schematic diagram of a training device for identifying a model of an elbow joint rotation axis provided by an embodiment of the present disclosure;

[0032] Figure 6 It is a structural schematic diagram of a device for identifying the rotation axis of an elbow joint provided by an embodiment of the present disclosure;

[0033] Figure 7 It is a structural schematic diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0034] In the following description, specific details such as specific system structures and technologies are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present disclosure. However, it should be clear to those skilled in the art that the present disclosure may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obstructing the description of the present disclosure with unnecessary details.

[0035] The following will describe in detail a training method, an identification method and a device for identifying an elbow joint rotation axis model according to an embodiment of the present disclosure in conjunction with the accompanying drawings.

[0036] Figure 1 FIG. 1 is a flow chart of a training method for an elbow joint rotation axis recognition model provided by an embodiment of the present disclosure. Figure 1 As shown, the training method of the elbow joint rotation axis recognition model includes:

[0037] Step S101, obtaining a three-dimensional image of a joint, wherein the three-dimensional image of the joint carries a joint rotation axis annotation marked by an annotator.

[0038] A 3D joint image is usually selected that carries at least two joint rotation axis annotations marked by the annotator.

[0039] In practical applications, it is usually difficult to obtain accurate annotations of the central axis of rotation of the elbow joint in non-standard anteroposterior and lateral two-dimensional images of the elbow joint. In order to solve the difficulty of directly collecting annotated two-dimensional images, the embodiment of the present disclosure uses three-dimensional image data as a basis to avoid the problem of difficult annotation of non-standard anteroposterior and lateral two-dimensional images, and collects annotated three-dimensional joint images as a training data set for subsequent steps. Three-dimensional joint images are widely used in current joint trauma surgery and are easy to collect. These three-dimensional joint images usually contain the intraoperative doctor's planning of the joint rotation center axis positioning needle, which can provide sufficient high-quality data for the subsequent training of the initial joint rotation axis recognition model, which is conducive to improving the efficiency of model training and model effect.

[0040] The three-dimensional image of the joint may be a three-dimensional CBCT image of the patient's traumatic joint area taken by a cone beam computer tomography (CBCT) machine before or during joint trauma surgery (such as elbow trauma surgery, etc.), or it may be a CT image or a three-dimensional image of other modalities.

[0041] In order to eliminate the subjective judgment deviation of the joint rotation axis (rotation center axis) caused by personal subjective factors to the greatest extent, it is preferably required that each joint three-dimensional image carries the joint rotation axis annotations made by at least two annotators (for example, doctor A and doctor B) for the three-dimensional image.

[0042] The joint rotation axis marking includes at least the positions of the entry point and the exit point of the rotation axis positioning needle marked by the annotator on the joint three-dimensional image. The rotation axis (axis) of the joint can be determined by the entry point and the exit point of the positioning needle.

[0043] Step S102, projecting the joint three-dimensional image and the joint rotation axis annotation it carries onto a preset two-dimensional image imaging plane to obtain a simulated two-dimensional image with the joint rotation axis annotation value.

[0044] In one embodiment, a CBCT projection matrix may be used to project the above-mentioned joint three-dimensional image together with the joint rotation axis mark carried by it onto a preset two-dimensional image imaging plane to obtain a simulated two-dimensional image. The CBCT projection matrix may be a projection matrix provided by a CBCT manufacturer.

[0045] The schematic diagram of the projection is as follows Figure 2 As shown, the "3D voxel" in the figure represents the three-dimensional image of the joint that needs to be projected, the tip of the vertebra represents the position of the radiation source (X-ray emission source), and the "DRR (Digitally Reconstructured Radiograph) plane" represents the two-dimensional image imaging plane.

[0046] Step S103, performing image modality conversion on the simulated two-dimensional image to obtain a converted two-dimensional image.

[0047] Generally, since the pixel value of a CBCT three-dimensional image is not equivalent to the attenuation coefficient of the photographed object, the simulated two-dimensional image obtained by this method and the real two-dimensional image obtained by directly photographing the photographed object (such as the elbow joint) by CBCT have obvious differences in contrast and other aspects. The model parameters obtained by simulating the two-dimensional image cannot be well applied to the reasoning of the real two-dimensional image. Therefore, it is necessary to convert the image modality of the simulated two-dimensional image into a converted two-dimensional image that is consistent with the image modality of the real two-dimensional image (that is, convert the image style of the simulated two-dimensional image into a converted two-dimensional image that is consistent with the image style of the real two-dimensional image), and then use it for subsequent reasoning to improve the accuracy and reliability of the reasoning results.

[0048] Step S104, using the series of two-dimensional images converted in step 103 and the elbow rotation axis corresponding to the series of images to input into a preset segmentation model, that is, training the initial joint rotation axis recognition model, until the preset training termination condition is reached, and obtaining the final joint rotation axis recognition model. The preset segmentation model can be a commonly used UNet model, etc.

[0049] The preset training termination condition can be set according to the actual situation. Conventionally, it can be that the number of training rounds reaches a preset round number threshold, or the model recognition accuracy reaches a preset accuracy threshold, etc.

[0050] The technical solution provided by the embodiment of the present disclosure is to obtain a three-dimensional joint image, which carries a joint rotation axis annotation marked by an annotator; project the three-dimensional joint image and the joint rotation axis annotation carried by it to a preset two-dimensional image imaging plane to obtain a simulated two-dimensional image; perform image modality conversion on the simulated two-dimensional image to obtain a converted two-dimensional image; use a series of two-dimensional images obtained by conversion and the elbow rotation axis corresponding to the series of images to input into a preset segmentation model until the preset training termination condition is reached to obtain a final joint rotation axis recognition model. The final joint rotation axis recognition model can accurately identify the rotation axis position of the joint image taken by the patient in any position, without strictly requiring the patient to place the joint in the standard elbow joint anteroposterior position during intraoperative filming, greatly reducing the difficulty of filming operation, and only one shot is required to achieve the requirement of accurately identifying the joint rotation axis, simplifying the surgical operation, shortening the time consumption, reducing intraoperative X-ray exposure, that is, reducing the radiation dose, and improving the accuracy, reliability and safety of the surgical results.

[0051] In some embodiments, the above step S102 includes:

[0052] Discretize the joint three-dimensional image to obtain a discretized detected object;

[0053] The X-rays having a preset incident intensity are emitted to the discretized detected object through a preset radiation source simulation;

[0054] A two-dimensional image imaging plane is used to receive the attenuated X-rays after the X-rays pass through the discretized detected object, and a simulated two-dimensional image is formed on the two-dimensional image imaging plane. The simulated two-dimensional image carries annotation points corresponding to the joint rotation axis annotation.

[0055] Combination Figure 2 , the DRR process actually simulates the imaging process of a two-dimensional image. Specifically, first, the three-dimensional image of the joint is discretized to obtain a discretized detected object; then, a preset radiation source (i.e., an X-ray emission source) is used to simulate the emission of X-rays with a preset incident intensity to the discretized detected object; then, the DRR plane (i.e., the two-dimensional image imaging plane) is used to receive the attenuated X-rays after the X-rays penetrate the discretized detected object, and a simulated two-dimensional image is formed on the two-dimensional image imaging plane, and the simulated two-dimensional image carries the annotation points corresponding to the above-mentioned joint rotation axis annotation. That is, the projection of the annotation points corresponding to the joint rotation axis annotation carried by the three-dimensional image of the joint will be displayed on the simulated two-dimensional image.

[0056] In the embodiment of the present disclosure, assuming that the incident intensity of the X-ray is I0, the residual intensity of the attenuated X-ray after passing through the discretized detected object is I, the attenuation coefficient of passing through the object is μ, and l represents the penetration length of the X-ray in the discretized detected object. Then, the relationship between the various variables satisfies the following formula (1).

[0057]

[0058] For the discretized detected object, the integral operation in the above formula (1) can be changed into an accumulation operation. In addition, since the attenuation coefficients of different positions of the detected object are different, in practical applications, the pixel values ​​of each position of the detected object can be used to approximately replace the attenuation coefficient of the position.

[0059] In the embodiment of the present disclosure, the projection angle of the joint three-dimensional image can be any angle, which is not limited here. Since the projection angle of the joint three-dimensional image can be any angle, the final joint rotation axis recognition model disclosed in the present disclosure has good robustness.

[0060] In some embodiments, the above step S103 includes:

[0061] Acquire a real two-dimensional image of the joint, wherein the imaging target of the real two-dimensional image is not required to be consistent with the target photographed by the three-dimensional image of the joint;

[0062] The simulated two-dimensional image and the real two-dimensional image are input into a preset recurrent adversarial generation network to convert the image modality of the simulated two-dimensional image into a converted two-dimensional image that is consistent with the image modality of the real two-dimensional image.

[0063] As an example, a real two-dimensional image of a photographed joint can be captured by CBCT. The photographed joint here can be the same joint as the photographed target of the joint three-dimensional image, or it can be a joint different from the photographed target of the joint three-dimensional image. For example, the joint three-dimensional image captures the joint of A's right hand, and the real two-dimensional image can be the joint of A's right hand or the joint of A's left hand.

[0064] Then, the simulated two-dimensional image and the real two-dimensional image obtained according to the above steps are input into a preset cyclic adversarial generative network (such as a CycleGAN network). Through the cyclic adversarial generative network, the image modality of the simulated two-dimensional image can be converted into a converted two-dimensional image that is consistent with the image modality of the real two-dimensional image while keeping the content contained in the simulated two-dimensional image unchanged.

[0065] Specific, combined Figure 3 , the cyclic adversarial generation network includes a first generator 301, a second generator 302, a first discriminator 303 and a second discriminator 304.

[0066] As an example, there are two groups of image samples with different image styles. Specifically in the present disclosure, the image samples of two different image modalities are respectively simulated two-dimensional images X and real two-dimensional images Y. The cyclic adversarial generative network can use the simulated two-dimensional images X and the real two-dimensional images Y to simultaneously train the first generator 301, the second generator 302, the first discriminator 303, and the second discriminator 304.

[0067] The specific training process is as follows: first, the simulated two-dimensional image X can be input into the first generator 301 to generate a first generated image y; then, the first generated image y is input into the first discriminator 303 to calculate the first similarity between the first generated image y and the real two-dimensional image Y; then, the first generated image y is input into the second generator 302 to generate a first regression image X', and the second similarity between the first regression image X' and the simulated two-dimensional image X is calculated. At the same time, the real two-dimensional image Y can be input into the second generator 302 to generate a second generated image x; the second generated image x is input into the second discriminator 304 to calculate the third similarity between the second generated image x and the simulated two-dimensional image X; the second generated image x is input into the first generator 301 to generate a second regression image Y'; and the fourth similarity between the second regression image Y' and the real two-dimensional image Y is calculated. When the first similarity, the second similarity, the third similarity and the fourth similarity meet the preset objective function, the converted two-dimensional image is output.

[0068] On the one hand, the model hopes that the samples generated by the generator can be as consistent as possible with the target samples and cannot be distinguished by the discriminator; on the other hand, the model also hopes that the discriminator function is as powerful as possible and can distinguish the samples generated by the generator and the target samples to the greatest extent.

[0069] Specifically in the above example, the first generator 301 can convert the simulated two-dimensional image X into the first generated image y, and the first discriminator 303 is used to distinguish the first generated image y from the real two-dimensional image Y. At the same time, in order to avoid the generated first generated image y losing the previous content, the first regression image X' can be further generated by the second generator 302. After the second generator 302, it can be regressed to the previously input simulated two-dimensional image X, and the difference between the first regression image X' and the simulated two-dimensional image X is as small as possible. The following formulas (2) and (3) need to be satisfied:

[0070] L adv (X→Y)=E y [logD y (y)]+E x [log(1-D y (G y (x)))] (2);

[0071] L cycle (X→Y→X)=E x [||G x (Gy(x))-x||] (3).

[0072] Among them, in formula (2) and (3), L adv (X→Y) represents the first similarity, E y [logD y (y)] represents the expected function that the first discriminator can distinguish the first generated image generated by the first generator from the real two-dimensional image to the greatest extent, E x [log(1-D y (G y (y))] then it is expected that the first generator (G y ) can be generated by the first generated image and the real two-dimensional image to the greatest extent possible without being identified by the first discriminator (D y ) to distinguish the expected function. cycle (X→Y→X) represents the second similarity, E x [||G x (G y (x))-x||] indicates that it is expected to pass the second generator (G x ) is a desired function that makes the difference between the first regression image generated and the simulated two-dimensional image X as small as possible.

[0073] Similarly, the second generator 302 can convert the real two-dimensional image Y into the second generated image x, and the second discriminator 304 is used to distinguish the second generated image x from the simulated two-dimensional image X. At the same time, in order to avoid the generated second generated image x losing the previous content, the second regression image Y' can be further generated by the first generator 301. After the first generator 301, it can be regressed to the real two-dimensional image Y input before, and the difference between the second regression image Y' and the real two-dimensional image Y is as small as possible. The following formulas (4) and (5) need to be satisfied:

[0074] L adv (Y→X)=E x [logD x (x)]+E y [log(1-D x (G x (y)))] (4);

[0075] L cycle (Y→X→Y)=E y [||G y (G x (y))-y||] (5).

[0076] Among them, in formula (4) and (5), L adv (Y→X) represents the third similarity, E X [logD x (x)] represents the expected function that the second discriminator can distinguish the second generated image generated by the second generator from the simulated two-dimensional image to the greatest extent, E y [log(1-D x (G x (y))] indicates that it is expected that the second generator (G x ) can be generated by the second generated image and the simulated two-dimensional image to the greatest extent possible without being detected by the second discriminator (D x ) to distinguish the expected function. cycle (Y→X→Y) represents the fourth similarity, E y [||G y (G x (y))-y||] indicates that it is expected to pass the first generator (G y ) is the desired function that the difference between the second regression image generated and the true two-dimensional image Y is as small as possible.

[0077] In some embodiments, when the first similarity, the second similarity, the third similarity, and the fourth similarity satisfy a preset objective function, outputting a converted two-dimensional image includes:

[0078] Calculate the product of the first similarity and the preset first loss coefficient to obtain a first loss value;

[0079] Calculate the product of the second similarity and a preset second loss coefficient to obtain a second loss value;

[0080] Calculate the product of the third similarity and the preset third loss coefficient to obtain a third loss value;

[0081] Calculate the product of the fourth similarity and the preset fourth loss coefficient to obtain a fourth loss value;

[0082] calculating a sum of a first loss value, a second loss value, a third loss value, and a fourth loss value;

[0083] If the sum reaches the preset target value, the converted two-dimensional image is output.

[0084] The first loss value L1 can be calculated according to the following formula (6).

[0085] L1=λ1*L adv (X→Y) (6).

[0086] Here, λ1 in equation (6) represents the first loss coefficient.

[0087] The second loss value L2 is calculated according to the following formula (7).

[0088] L2=λ2*L cycle (X→Y→X) (7).

[0089] Here, λ2 in equation (7) represents the second loss coefficient.

[0090] The third loss value L3 is calculated according to the following formula (8).

[0091] L3=λ3*L adv (Y→X) (8).

[0092] Here, λ3 in equation (8) represents the third loss coefficient.

[0093] The fourth loss value L4 is calculated according to the following formula (9).

[0094] L4=λ4*L cycle (Y→X→Y) (9).

[0095] Here, λ4 in equation (9) represents the fourth loss coefficient.

[0096] In order to simultaneously satisfy the following requirements: ① the samples generated by the generator can be as consistent as possible with the target samples without being distinguished by the discriminator; ② the discriminator can distinguish the samples generated by the generator from the target samples to the greatest extent, we can jointly construct the objective function L(G) as shown in the following formula (10): x ,Dx ,G y ,D y ).

[0097] L(G x ,D x ,G y ,D y )=L1+L2+L3+L4 (10).

[0098] If the function value of the above objective function reaches a preset target value, for example, the sum of the first loss value, the second loss value, the third loss value and the fourth loss value is the smallest, then the converted two-dimensional image is output.

[0099] It should be noted that the first, second, third and fourth loss coefficients mentioned above can be adjusted according to actual conditions. For example, they can all be set to 1.

[0100] In some embodiments, the above step S104 includes:

[0101] Extracting and converting the marked points of the two-dimensional image, the marked points include a first planning positioning point and a second planning positioning point;

[0102] The converted two-dimensional image, the first planning positioning point and the second planning positioning point are trained and input into a preset segmentation model, namely, an initial rotation axis recognition model, until a preset training termination condition is reached to obtain a final rotation axis recognition model.

[0103] Specifically, the first planning positioning point (positioning needle entry point) and the second planning positioning point (positioning needle exit point) of each converted two-dimensional image are extracted. Then, the first planning positioning point is used as the first sphere center, and the sphere center is expanded into a first spherical structure with a first preset radius, and the second planning positioning point is used as the second sphere center, and the sphere center is expanded into a second spherical structure with a second preset radius; and the converted two-dimensional image, the first spherical structure and the second spherical structure are used to train the initial rotation axis recognition model.

[0104] The first preset radius and the second preset radius can be flexibly set according to actual conditions, for example, they can be set to 2 mm, 3 mm, etc. The first preset radius and the second preset radius can be set to the same value or different values. Preferably, they are set to the same value.

[0105] By extracting the first planned positioning point and the second planned positioning point for determining the rotation axis in each converted two-dimensional image, and expanding them into a first spherical structure and a second spherical structure with a certain radius, respectively, the problem of finding feature points can be converted into an image segmentation problem, thereby simplifying the model training process. The initial rotation axis recognition model can be an image segmentation network (such as a UNet network). The UNet network is the most widely used and effective network model in the field of image segmentation. Therefore, the present invention directly inputs the converted two-dimensional image and the first spherical structure and the second spherical structure obtained in the above steps into the UNet network for training until the preset training end conditions are reached, thereby obtaining the final joint rotation axis recognition model.

[0106] Figure 4 FIG. 1 is a flow chart of a method for identifying the rotation axis of an elbow joint provided by an embodiment of the present disclosure. Figure 4 As shown, the elbow joint rotation axis identification method includes:

[0107] Step S401, obtaining a perspective image of a joint to be identified;

[0108] Step S402, inputting the joint perspective image to be identified into the final joint rotation axis recognition model obtained above, and outputting the rotation axis recognition result of the joint perspective image.

[0109] As an example, assuming that the acquired joint perspective image to be identified is an elbow joint perspective image (for example, one intraoperative perspective image of the elbow joint in the near-anteroposterior and lateral positions), then the elbow joint perspective image can be input into the above-mentioned final joint rotation axis recognition model to complete the recognition of the elbow joint rotation axis and output the rotation axis recognition result.

[0110] The technical solution provided by the disclosed embodiment can quickly identify the rotation axis position of the joint by inputting the acquired joint fluoroscopic image into the above-mentioned final joint rotation axis recognition model, without the need for the patient to cooperate in strictly placing the joint position in the positive or lateral position, thereby simplifying the difficulty of taking films during surgery. There is no need to repeat the film, only one film is needed, which greatly reduces the intraoperative X-ray exposure. At the same time, it can make the doctor no longer need to focus on marking the fluoroscopic image, so that the doctor can focus more on the operation itself, which is conducive to eliminating the influence of the doctor's subjective factors and random factors on the operation results, and is conducive to improving the safety of the operation, reducing the operation time, and improving the accuracy of the operation.

[0111] In one embodiment, the technical solution provided by the embodiment of the present disclosure can be applied to a robotic surgery system, which includes a robotic arm, a host computer, a cone beam computed tomography scanner, an optical tracking system, a positioning ruler, and a data processing center loaded with the above-mentioned elbow joint rotation axis recognition program. The data processing center can use the cone beam computed tomography scanner to scan the patient's joint three-dimensional image, complete the recognition of the joint rotation axis, and output the recognition result. The recognition result can be further used to assist doctors in surgical path planning, which is conducive to improving the accuracy, reliability and safety of the operation.

[0112] Exemplarily, the surgical process of positioning the rotation axis of the elbow joint under robot positioning is as follows: first, arrange the environment according to the surgical operation process instructions, and install the positioning ruler, tracker, etc.; then, collect one intraoperative fluoroscopic image each of the elbow joint in the near-anteroposterior position and the lateral position; then, perform image registration on these two fluoroscopic images; thereafter, input the registered images into the final joint rotation axis recognition model disclosed in the present invention, identify the central axis of the elbow joint rotation, generate the surgical plan, and guide and assist the doctor to complete the operation accurately and quickly.

[0113] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present application, which will not be described one by one here.

[0114] The following are embodiments of the device disclosed herein, which can be used to execute the method embodiments disclosed herein. For details not disclosed in the device embodiments disclosed herein, please refer to the method embodiments disclosed herein.

[0115] Figure 5 Schematic diagram of a training device for identifying an elbow joint rotation axis model provided by an embodiment of the present disclosure. Figure 5 As shown, the training device of the elbow joint rotation axis recognition model includes:

[0116] An image acquisition module 501 is configured to acquire a three-dimensional image of a joint, wherein the three-dimensional image of the joint carries a joint rotation axis annotation marked by an annotator;

[0117] The image simulation module 502 is configured to project the joint three-dimensional image and the joint rotation axis annotation carried by it onto a preset two-dimensional image imaging plane to obtain a simulated two-dimensional image;

[0118] The modality conversion module 503 is configured to perform image modality conversion on the simulated two-dimensional image to obtain a converted two-dimensional image;

[0119] The model training module 504 is configured to use the converted two-dimensional image to train the initial joint rotation axis recognition model until a preset training termination condition is reached to obtain a final joint rotation axis recognition model.

[0120] The technical solution provided by the embodiment of the present disclosure is that the joint three-dimensional image is acquired by the image acquisition module 501, and the joint three-dimensional image carries the joint rotation axis annotation marked by the annotator; the image simulation module 502 projects the joint three-dimensional image and the joint rotation axis annotation carried by it to the preset two-dimensional image imaging plane to obtain a simulated two-dimensional image; the modal conversion module 503 performs image modal conversion on the simulated two-dimensional image to obtain a converted two-dimensional image; the model training module 504 uses the converted two-dimensional image to train the initial joint rotation axis recognition model until the preset training termination condition is reached to obtain the final joint rotation axis recognition model. The final joint rotation axis recognition model can accurately identify the rotation axis position of the joint image taken by the patient in any position, without strictly requiring the patient to place the joint in the standard elbow joint lateral position during the intraoperative filming, which greatly reduces the difficulty of the filming operation, and only one shot is required to achieve the requirement of accurately identifying the joint rotation axis, simplifying the surgical operation, shortening the time consumption, reducing intraoperative X-ray exposure, that is, reducing the radiation dose, and improving the accuracy, reliability and safety of the surgical results.

[0121] In some embodiments, the image simulation module 502 includes:

[0122] A discretization unit is configured to discretize the three-dimensional image of the joint to obtain a discretized detected object;

[0123] A transmitting unit, configured to simulate the emission of X-rays of a preset incident intensity to the discretized detected object through a preset radiation source;

[0124] The generating unit is configured to use a two-dimensional image imaging plane to receive the attenuated X-rays after the X-rays pass through the discretized detected object, and to form a simulated two-dimensional image on the two-dimensional image imaging plane, wherein the simulated two-dimensional image carries annotation points corresponding to the annotations of the joint rotation axis.

[0125] In some embodiments, the above-mentioned mode conversion module 503 includes:

[0126] A real image acquisition unit is configured to acquire a real two-dimensional image of the joint, wherein an imaging target of the real two-dimensional image is not required to be consistent with a target photographed by the three-dimensional image of the joint;

[0127] The conversion unit is configured to input the simulated two-dimensional image and the real two-dimensional image into a preset recurrent adversarial generation network to convert the image modality of the simulated two-dimensional image into a converted two-dimensional image consistent with the image modality of the real two-dimensional image.

[0128] In some embodiments, the above conversion unit may be specifically configured as follows:

[0129] Inputting the simulated two-dimensional image into the first generator to generate a first generated image;

[0130] Inputting the first generated image into a first discriminator, and calculating a first similarity between the first generated image and the real two-dimensional image;

[0131] Inputting the first generated image into the second generator to generate a first regression image;

[0132] calculating a second similarity between the first regression image and the simulated two-dimensional image;

[0133] Inputting the real two-dimensional image into the second generator to generate a second generated image;

[0134] inputting the second generated image into a second discriminator, and calculating a third similarity between the second generated image and the simulated two-dimensional image;

[0135] Inputting the second generated image into the first generator to generate a second regression image;

[0136] Calculating a fourth similarity between the second regression image and the true two-dimensional image;

[0137] When the first similarity, the second similarity, the third similarity and the fourth similarity satisfy a preset objective function, a converted two-dimensional image is output.

[0138] In some embodiments, when the first similarity, the second similarity, the third similarity, and the fourth similarity satisfy a preset objective function, outputting a converted two-dimensional image includes:

[0139] Calculate the product of the first similarity and the preset first loss coefficient to obtain a first loss value;

[0140] Calculate the product of the second similarity and a preset second loss coefficient to obtain a second loss value;

[0141] Calculate the product of the third similarity and the preset third loss coefficient to obtain a third loss value;

[0142] Calculate the product of the fourth similarity and the preset fourth loss coefficient to obtain a fourth loss value;

[0143] calculating a sum of a first loss value, a second loss value, a third loss value, and a fourth loss value;

[0144] If the sum reaches the preset target value, the converted two-dimensional image is output.

[0145] In some embodiments, the model training module 504 includes:

[0146] An extraction unit is configured to extract the annotation points of the converted two-dimensional image, where the annotation points include a first planning positioning point and a second planning positioning point;

[0147] The training unit is configured to train an initial rotation axis recognition model using the converted two-dimensional image, the first planning positioning point and the second planning positioning point until a preset training termination condition is reached to obtain a final rotation axis recognition model.

[0148] In some embodiments, the training unit may be specifically configured as follows:

[0149] Taking the first planned positioning point as the first sphere center, the sphere center is expanded into a first spherical structure with a first preset radius, and taking the second planned positioning point as the second sphere center, the sphere center is expanded into a second spherical structure with a second preset radius;

[0150] An initial rotation axis recognition model is trained using the transformed two-dimensional image, the first spherical structure, and the second spherical structure.

[0151] Figure 6 Schematic diagram of a training device for identifying an elbow joint rotation axis model provided by an embodiment of the present disclosure. Figure 6 As shown, the elbow joint rotation axis identification device comprises:

[0152] An acquisition module 601 is configured to acquire a perspective image of a joint to be identified;

[0153] The image recognition module 602 is configured to input the joint perspective image to be recognized into the above-mentioned final joint rotation axis recognition model, and output the rotation axis recognition result of the joint perspective image.

[0154] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present disclosure.

[0155] Figure 7 Schematic diagram of an electronic device 7 provided by an embodiment of the present disclosure. Figure 7 As shown, the electronic device 7 of this embodiment includes: a processor 701, a memory 702, and a computer program 703 stored in the memory 702 and executable on the processor 701. When the processor 701 executes the computer program 703, the steps in the above-mentioned method embodiments are implemented. Alternatively, when the processor 701 executes the computer program 703, the functions of the modules / units in the above-mentioned device embodiments are implemented.

[0156] The electronic device 7 may be a desktop computer, a notebook, a PDA, a cloud server, or other electronic device. The electronic device 7 may include, but is not limited to, a processor 701 and a memory 702. Those skilled in the art will appreciate that Figure 7 The electronic device 7 is merely an example and does not limit the electronic device 7 , and may include more or less components than those shown in the figure, or different components.

[0157] The processor 701 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0158] The memory 702 may be an internal storage unit of the electronic device 7, for example, a hard disk or memory of the electronic device 7. The memory 702 may also be an external storage device of the electronic device 7, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 7. The memory 702 may also include both an internal storage unit of the electronic device 7 and an external storage device. The memory 702 is used to store computer programs and other programs and data required by the electronic device.

[0159] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units.

[0160] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present disclosure implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. The computer program may include computer program code, and the computer program code may be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electric carrier signals and telecommunication signals.

[0161] The above embodiments are only used to illustrate the technical solutions of the present disclosure, rather than to limit them. Although the present disclosure has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should all be included in the protection scope of the present disclosure.

Claims

1. A training method for an elbow joint rotation axis recognition model, characterized in that: include: Acquire a three-dimensional image of the joint, wherein the three-dimensional image of the joint carries a joint rotation axis annotation marked by an annotator; Projecting the joint three-dimensional image and the joint rotation axis mark carried by it onto a preset two-dimensional image imaging plane to obtain a simulated two-dimensional image; Performing image modality conversion on the simulated two-dimensional image to obtain a converted two-dimensional image; Using the converted two-dimensional image to train an initial joint rotation axis recognition model until a preset training termination condition is reached to obtain a final joint rotation axis recognition model; The method of using the converted two-dimensional image to train the initial joint rotation axis recognition model until a preset training termination condition is reached to obtain a final joint rotation axis recognition model includes: Extracting the marked points of the converted two-dimensional image, wherein the marked points include a first planned positioning point and a second planned positioning point; Using the converted two-dimensional image, the first planned positioning point and the second planned positioning point to train an initial rotation axis recognition model until a preset training termination condition is reached to obtain a final rotation axis recognition model; The method of using the converted two-dimensional image, the first planning positioning point and the second planning positioning point to train an initial rotation axis recognition model includes: Taking the first planned positioning point as the first sphere center, the sphere center is expanded into a first spherical structure with a first preset radius; taking the second planned positioning point as the second sphere center, the sphere center is expanded into a second spherical structure with a second preset radius; An initial rotation axis recognition model is trained using the converted two-dimensional image, the first spherical structure, and the second spherical structure.

2. The method according to claim 1, characterized in that: The three-dimensional joint image and the joint rotation axis mark carried by the image are projected onto a preset two-dimensional image imaging plane to obtain a simulated two-dimensional image, including: Discretizing the joint three-dimensional image to obtain a discretized detected object; Simulating emitting X-rays of a preset incident intensity to the discretized detected object through a preset radiation source; The two-dimensional image imaging plane is used to receive the attenuated X-rays after the X-rays pass through the discretized detected object, and a simulated two-dimensional image is formed on the two-dimensional image imaging plane, wherein the simulated two-dimensional image carries annotation points corresponding to the annotation of the joint rotation axis.

3. The method according to claim 1, characterized in that Performing image modality conversion on the simulated two-dimensional image to obtain a converted two-dimensional image includes: Acquire a real two-dimensional image of the joint, wherein the imaging target of the real two-dimensional image is not required to be consistent with the target photographed by the three-dimensional image of the joint; The simulated two-dimensional image and the real two-dimensional image are input into a preset recurrent adversarial generation network to convert the image modality of the simulated two-dimensional image into a converted two-dimensional image that is consistent with the image modality of the real two-dimensional image.

4. The method according to claim 3, characterized in that The cyclic adversarial generative network includes a first generator, a second generator, a first discriminator and a second discriminator; Inputting the simulated two-dimensional image and the real two-dimensional image into a preset recurrent adversarial generative network to convert the image modality of the simulated two-dimensional image into a converted two-dimensional image consistent with the image modality of the real two-dimensional image, comprising: Inputting the simulated two-dimensional image into the first generator to generate a first generated image; Inputting the first generated image into the first discriminator, and calculating a first similarity between the first generated image and the real two-dimensional image; Inputting the first generated image into the second generator to generate a first regression image; calculating a second similarity between the first regression image and the simulated two-dimensional image; Inputting the real two-dimensional image into the second generator to generate a second generated image; inputting the second generated image into the second discriminator, and calculating a third similarity between the second generated image and the simulated two-dimensional image; Inputting the second generated image into the first generator to generate a second regression image; Calculating a fourth similarity between the second regression image and the real two-dimensional image; When the first similarity, the second similarity, the third similarity and the fourth similarity satisfy a preset objective function, a converted two-dimensional image is output.

5. The method according to claim 4, characterized in that When the first similarity, the second similarity, the third similarity and the fourth similarity satisfy a preset objective function, outputting a converted two-dimensional image comprises: Calculate the product of the first similarity and a preset first loss coefficient to obtain a first loss value; Calculate the product of the second similarity and a preset second loss coefficient to obtain a second loss value; Calculating the product of the third similarity and a preset third loss coefficient to obtain a third loss value; Calculating the product of the fourth similarity and a preset fourth loss coefficient to obtain a fourth loss value; Calculating the sum of the first loss value, the second loss value, the third loss value and the fourth loss value; If the sum obtains a preset target value, a converted two-dimensional image is output.

6. A method for identifying the rotation axis of an elbow joint, characterized in that: include: Acquire a perspective image of a joint to be identified; The joint perspective image to be identified is input into the elbow joint rotation axis identification model as described in claim 1, and the rotation axis identification result of the joint perspective image is output.

7. A training device for identifying a model of an elbow joint rotation axis, characterized in that: include: An image acquisition module is configured to acquire a three-dimensional image of a joint, wherein the three-dimensional image of the joint carries at least two joint rotation axis annotations marked by annotators; An image simulation module is configured to project the joint three-dimensional image and the joint rotation axis annotation carried by it onto a preset two-dimensional image imaging plane to obtain a simulated two-dimensional image; a modality conversion module, configured to perform image modality conversion on the simulated two-dimensional image to obtain a converted two-dimensional image; The model training module is configured to use the converted two-dimensional image to train the initial joint rotation axis recognition model until a preset training termination condition is reached to obtain a final joint rotation axis recognition model. The method of using the converted two-dimensional image to train the initial joint rotation axis recognition model until a preset training termination condition is reached to obtain the final joint rotation axis recognition model includes: extracting the annotated points of the converted two-dimensional image, wherein the annotated points include a first planning positioning point and a second planning positioning point; using the converted two-dimensional image, the first planning positioning point and the second planning positioning point to train the initial rotation axis recognition model until a preset training termination condition is reached to obtain the final rotation axis recognition model; using the converted two-dimensional image, the first planning positioning point and the second planning positioning point to train the initial rotation axis recognition model includes: taking the first planning positioning point as the first spherical center, expanding the spherical center into a first spherical structure with a first preset radius, taking the second planning positioning point as the second spherical center, expanding the spherical center into a second spherical structure with a second preset radius; using the converted two-dimensional image, the first spherical structure and the second spherical structure to train the initial rotation axis recognition model.

8. A device for identifying the rotation axis of an elbow joint, characterized in that: include: An acquisition module, configured to acquire a perspective image of a joint to be identified; The image recognition module is configured to input the joint perspective image to be recognized into the elbow joint rotation axis recognition model as described in claim 1, and output the rotation axis recognition result of the joint perspective image.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

11. A robotic surgery system, comprising a robotic arm, a host computer, a cone beam computed tomography scanner, an optical tracking system, and a positioning scale, characterized in that: It also includes the elbow joint rotation axis identification device as described in claim 8.

Images