Knee joint surgical navigation positioning method and system based on neural network combined medical image

Abstract

The invention relates to the technical field of surgical navigation and positioning, in particular to a knee joint surgical navigation and positioning method and system based on neural network and medical image.The method comprises the steps that a surgical printing model is constructed, three-dimensional scanning is conducted on the surgical printing model, a three-dimensional knee joint image is obtained, two-dimensional scanning is conducted on the surgical printing model, and an azimuth knee joint image set is obtained; performing automatic registration on the orientation knee joint image set and the three-dimensional knee joint image to obtain a training registration parameter set, training a plurality of neural network models according to the training registration parameter set to obtain a plurality of adaptive registration models, and integrating the adaptive registration models to obtain an integrated registration model, and inputting the intraoperative two-dimensional image and the three-dimensional knee joint image into the integrated registration model to obtain a three-dimensional navigation positioning image. According to the method, the accuracy of navigation positioning in the knee joint operation can be improved, and the image registration time in the knee joint operation is shortened.

Description

Technical Field

[0001] The present invention relates to the technical field of surgical navigation and positioning, and particularly to a knee joint surgical navigation and positioning method and system based on neural network combined with medical images. Background Art

[0002] In knee joint surgery, the accuracy of surgical navigation and positioning is crucial for the success of the surgery. Accurate navigation can not only help surgeons perform surgical operations more precisely, reduce surgical risks and complications, but also improve surgical efficiency and shorten the recovery time of patients. Therefore, developing a technology that can provide real-time and high-precision navigation and positioning is of great significance for improving the overall quality of knee joint surgery and the treatment effect of patients.

[0003] Existing knee joint surgical navigation and positioning technologies mainly rely on traditional multimodal registration methods, such as feature point-based registration, mutual information-based registration, etc. These methods require a long calculation time to complete image registration, which causes a large delay in intraoperative real-time navigation. Moreover, when dealing with complex knee joint anatomical structures, these methods often require manual intervention to adjust registration parameters, increasing the complexity and uncertainty of the surgery, thus reducing the accuracy of navigation and positioning during the surgery. Summary of the Invention

[0004] The present invention provides a knee joint surgical navigation and positioning method and system based on neural network combined with medical images, and its main purpose is to improve the accuracy of navigation and positioning in knee joint surgery and reduce the time of image registration in knee joint surgery.

[0005] To achieve the above object, a knee joint surgical navigation and positioning method based on neural network combined with medical images provided by the present invention includes: Receiving a knee joint surgery instruction, determining a patient to be operated on based on the knee joint surgery instruction, and obtaining information of the patient to be operated on, wherein the information of the patient to be operated on includes: basic patient information, clinical diagnosis information, and surgical plan information; Constructing a surgical printing model according to the information of the patient to be operated on, performing three-dimensional scanning on the surgical printing model to obtain a three-dimensional knee joint image, wherein the surgical printing model includes a set of marker balls; Performing two-dimensional scanning on the surgical printing model to obtain a set of azimuth knee joint images, wherein the set of azimuth knee joint images includes multiple azimuth knee joint images, and one marker ball in the set of marker balls is included in the azimuth knee joint image; Sequentially extracting azimuth knee joint images from the set of azimuth knee joint images, and automatically registering the azimuth knee joint images and the three-dimensional knee joint image by using a preset multimodal registration system to obtain training registration parameters; Summarize the training registration parameters to obtain a training registration parameter set, and train multiple pre-acquired neural network models according to the training registration parameter set to obtain multiple adaptive registration models; Perform model error analysis on multiple adaptive registration models to obtain multiple registration error values and multiple optimal registration marker balls; Integrate the multiple adaptive registration models according to the multiple registration error values and multiple optimal registration marker balls to obtain an integrated registration model; Obtain intraoperative two-dimensional images, input the intraoperative two-dimensional images and three-dimensional knee joint images into the integrated registration model to obtain three-dimensional navigation positioning images, and complete knee joint surgery navigation positioning based on the neural network combined with medical images based on the three-dimensional navigation positioning images.

[0006] Optionally, the constructing a surgical printing model according to the information of the patient to be operated on includes: Print an initial printing model according to the information of the patient to be operated on; Identify the marked part group in the information of the patient to be operated on, obtain the marked positions of each marked part in the marked part group to obtain a marked position group; Obtain a marker ball group based on the marked part group, and inlay the marker ball group on the initial printing model according to the marked position group to obtain a surgical printing model.

[0007] Optionally, the performing two-dimensional scanning on the surgical printing model to obtain an azimuth knee joint image set includes: Extract marker balls from the marker ball group in sequence, set an initial shooting direction according to the marker balls, and determine the surgical shooting method of the patient to be operated on; Perform two-dimensional shooting on the surgical printing model based on the surgical shooting method, the initial shooting direction and the preset single-azimuth shooting accuracy to obtain an original azimuth image group, where the number of original azimuth images in the original azimuth image group is the same as the single-azimuth shooting accuracy; Change the shooting azimuth of the initial shooting direction to obtain a changed shooting direction, where the shooting azimuth change refers to translating or rotating the initial shooting direction; Take the changed shooting direction as the initial shooting direction, and return to the step of performing two-dimensional shooting on the surgical printing model based on the surgical shooting method, the initial shooting direction and the preset single-azimuth shooting accuracy until the number of the original azimuth image group is not less than the preset minimum number of image groups; Merge the original azimuth image group to obtain an azimuth knee joint image set.

[0008] Optionally, the performing model error analysis on multiple adaptive registration models to obtain multiple registration error values and multiple optimal registration marker balls includes: Select a test orientation image group from the azimuth knee joint image set, and pair each test orientation image in the test orientation image group with the three-dimensional knee joint image to obtain multiple test knee joint image groups. Among them, each test knee joint image group includes a test orientation image and a three-dimensional knee joint image; Extract the test knee joint image groups in sequence from the multiple test knee joint image groups, and perform the following operations on the test knee joint image groups: Extract the adaptive registration models in sequence from the multiple adaptive registration models, input the test knee joint image group into the adaptive registration model, and obtain test registration parameters. Among them, the test registration parameters include: rotation matrix and translation vector; Identify the three-dimensional center coordinate and two-dimensional center coordinate in the test knee joint image group, and calculate the unit error value based on the test registration parameters, three-dimensional center coordinate and two-dimensional center coordinate; Summarize the unit error values to obtain a unit error value group. Based on the unit error value group, determine the registration error value and the optimal registration marker ball, and summarize the registration error value and the optimal registration marker ball respectively to obtain multiple registration error values and multiple optimal registration marker balls.

[0009] Optionally, the identifying the three-dimensional center coordinate and two-dimensional center coordinate in the test knee joint image group includes: Confirm the test three-dimensional image and test two-dimensional image in the test knee joint image group; Obtain the binary image of the test three-dimensional image, perform connected component operation on the binary image to obtain a connected component group; Determine the pixel volume of the marker ball, set the marker ball volume range according to the pixel volume, and filter the connected component group by using the marker ball volume range to obtain an effective connected component group. Among them, the effective connected component group includes multiple effective connected components, and the volume of the effective connected component is within the marker ball volume range; Extract the effective connected components in sequence from the effective connected component group, determine the boundary image of the effective connected component, and perform three-dimensional Hough transform on the boundary image to obtain a transformed image; Identify the maximum pixel point in the transformed image, where the maximum pixel point is the pixel point with the largest pixel value in the transformed image; Summarize the maximum pixel points to obtain a maximum pixel point group, and determine the marker ball pixel point group in the maximum pixel point group based on the preset number of marker balls. Among them, the number of marker ball pixel points in the marker ball pixel point group is the same as the number of marker balls; Obtain the marker ball coordinate group of the marker ball pixel point group; Determine the two-dimensional marker ball and two-dimensional center coordinate in the test two-dimensional image, and determine the three-dimensional center coordinate corresponding to the two-dimensional marker ball in the marker ball coordinate group.

[0010] Optionally, calculating a unit error value based on the test registration parameters, three-dimensional sphere center coordinates, and two-dimensional sphere center coordinates includes: Identifying a two-dimensional photographing device and constructing an internal parameter matrix of the two-dimensional photographing device, where the internal parameter matrix is expressed as: Where, represents the internal parameter matrix, and respectively represent the focal lengths of the two-dimensional photographing device on the x-axis and y-axis, and respectively represent the abscissa and ordinate in the two-dimensional sphere center coordinates; Calculating transformed three-dimensional coordinates based on the test registration parameters, internal parameter matrix, and three-dimensional sphere center coordinates, where the transformed three-dimensional coordinates are expressed as: Where, represents the transformed three-dimensional coordinates, represents the abscissa in the transformed three-dimensional coordinates, represents the ordinate in the transformed three-dimensional coordinates, represents the vertical coordinate in the transformed three-dimensional coordinates, represents the rotation matrix in the test registration parameters, represents the three-dimensional sphere center coordinates, represents the abscissa in the three-dimensional sphere center coordinates, represents the ordinate in the three-dimensional sphere center coordinates, represents the vertical coordinate in the three-dimensional sphere center coordinates, represents the translation vector in the test registration parameters, represents the product symbol of vectors; Calculating the unit error value based on the transformed three-dimensional coordinates and the two-dimensional sphere center coordinates.

[0011] Optionally, calculating the unit error value based on the transformed three-dimensional coordinates and the two-dimensional sphere center coordinates includes: Performing a two-dimensional projection on the transformed three-dimensional coordinates to obtain two-dimensional transformed coordinates, where the two-dimensional transformed coordinates are expressed as: Where, represents the two-dimensional transformed coordinates, represents the abscissa in the two-dimensional transformed coordinates, represents the ordinate in the two-dimensional transformed coordinates; Calculating the unit error value according to the two-dimensional transformed coordinates and the two-dimensional sphere center coordinates using the following formula: Where, represents the unit error value.

[0012] Optionally, determining a registration error value and an optimal registration marker sphere based on the unit error value group includes: Identify the error marker balls corresponding to the unit error values in the unit error value group to obtain an error marker ball group, where the error marker ball is the marker ball corresponding to the two-dimensional spherical center coordinates in the formula for calculating the unit error value; Classify the unit error value group according to the error marker ball group to obtain multiple groups of similar error values. Among them, in the same group of similar error values, all the similar error values correspond to the same error marker ball; Set the marker ball weight group of the error marker ball group, calculate the average similar error value of each group of similar error values in the multiple groups of similar error values, and obtain an average similar error value group; Calculate the registration error value according to the marker ball weight group and the average similar error value group, where the registration error value is expressed as: Where, represents the registration error value, represents the number of groups of similar error values in the multiple groups of similar error values, represents the th marker ball weight in the marker ball weight group, represents the th average similar error value in the average similar error value group; Identify the optimal error value in the average similar error value group, where the optimal error value is the average similar error value with the smallest value in the average similar error value group; Identify the optimal registration marker ball corresponding to the optimal error value in the error marker ball group.

[0013] Optionally, the integration of the multiple adaptive registration models to obtain an integrated registration model includes: Extract the adaptive registration models in the multiple adaptive registration models in sequence, and identify the model error value of the adaptive registration model among the multiple registration error values; Confirm the optimal registration marker ball of the adaptive registration model, and identify the optimal registration weight corresponding to the optimal registration marker ball in the marker ball weight group; Calculate the model integration weight according to the model error value and the optimal registration weight by using the following formula: Where, represents the model integration weight, represents the optimal registration weight, represents the natural constant, represents the model error value, represents the th registration error value among the multiple registration error values, represents the number of registration error values among the multiple registration error values; Summarize the integrated weights of the models to obtain the integrated weights of multiple models. According to the integrated weights of the multiple models, integrate multiple adaptive registration models to obtain an integrated registration model.

[0014] To achieve the above object, the present invention also provides a knee joint surgical navigation and positioning system based on a neural network combined with medical images, including: A three-dimensional image scanning module, configured to receive a knee joint surgery instruction, determine a patient to be operated on based on the knee joint surgery instruction, and obtain information of the patient to be operated on, where the information of the patient to be operated on includes: basic patient information, clinical diagnosis information, and surgical plan information. Construct a surgical printing model according to the information of the patient to be operated on, perform three-dimensional scanning on the surgical printing model to obtain a three-dimensional knee joint image, where the surgical printing model includes a set of marker balls; A training parameter acquisition module, configured to perform two-dimensional scanning on the surgical printing model to obtain a set of azimuth knee joint images, where the set of azimuth knee joint images includes multiple azimuth knee joint images, and one marker ball in the set of marker balls is included in the azimuth knee joint image. Sequentially extract the azimuth knee joint images in the set of azimuth knee joint images, and use a preset multi-modal registration system to automatically register the azimuth knee joint images and the three-dimensional knee joint images to obtain training registration parameters; A model error analysis module, configured to summarize the training registration parameters to obtain a set of training registration parameters, train a plurality of pre-acquired neural network models according to the set of training registration parameters to obtain a plurality of adaptive registration models, and perform model error analysis on all the plurality of adaptive registration models to obtain a plurality of registration error values and a plurality of optimal registration marker balls; An intraoperative image registration module, configured to integrate the plurality of adaptive registration models according to the plurality of registration error values and the plurality of optimal registration marker balls to obtain an integrated registration model, obtain an intraoperative two-dimensional image, and input the intraoperative two-dimensional image and the three-dimensional knee joint image into the integrated registration model to obtain a three-dimensional navigation and positioning image.

[0015] To solve the above problems, the present invention also provides an electronic device, where the electronic device includes: A memory, storing at least one instruction; A processor, executing the instruction stored in the memory to implement the above-mentioned knee joint surgical navigation and positioning method based on a neural network combined with medical images.

[0016] To solve the above problems, the present invention also provides a computer-readable storage medium, where at least one instruction is stored in the computer-readable storage medium, and the at least one instruction is executed by a processor in an electronic device to implement the above-mentioned knee joint surgical navigation and positioning method based on a neural network combined with medical images.

[0017] To solve the problems described in the background art, the present invention constructs a surgical printing model and performs three-dimensional scanning, which can generate high-precision three-dimensional knee joint images, providing an intuitive three-dimensional anatomical structure for surgical navigation. The marker ball group on the surgical printing model provides accurate reference points for subsequent image registration, ensuring that the two-dimensional images can be accurately aligned to the three-dimensional images. Then, by two-dimensionally scanning the surgical printing model, multi-directional knee joint images can be obtained. These multi-directional knee joint images show the anatomical structure of the knee joint from different angles, thus providing diverse data for the training of the subsequent neural network model, improving the generalization ability and robustness of the model. Further, using a multi-modal registration system, the multi-directional knee joint images and the three-dimensional knee joint images are automatically registered to obtain training registration parameters. These training registration parameters provide key data support for the training of the neural network model. Multiple neural network models are trained according to the training registration parameter set to obtain multiple adaptive registration models. These adaptive registration models can automatically adapt to different image features and transformation relationships by learning a large number of registration parameters, thus quickly and accurately completing image registration during the operation. The training of multiple models not only improves the registration accuracy but also enhances the robustness of the system, enabling it to handle complex clinical scenarios and individual differences. Then, through model error analysis, the performance of each adaptive registration model is evaluated to obtain the registration error value and the optimal registration marker ball of each adaptive registration model. These registration error values and optimal registration marker balls provide important reference bases for subsequent model integration. By integrating multiple adaptive registration models, the advantages of each model can be combined to generate an integrated registration model with better performance. The integrated registration model can make full use of the registration advantages of each adaptive registration model in different knee joint parts, reduce the limitations of a single model, and improve the overall registration accuracy and robustness, thereby significantly improving the accuracy and reliability of surgical navigation. Finally, the intraoperative two-dimensional images and the three-dimensional knee joint images are input into the integrated registration model, and three-dimensional navigation positioning images can be quickly generated, providing real-time navigation information for the surgeon and greatly reducing the time for intraoperative image registration. Therefore, the present invention can improve the accuracy of navigation positioning in knee joint surgery and reduce the time for image registration in knee joint surgery. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a schematic flowchart of a knee joint surgical navigation and positioning method based on a neural network combined with medical images provided by an embodiment of the present invention; Figure 2 It is a functional module diagram of a knee joint surgical navigation and positioning system based on a neural network combined with medical images provided by an embodiment of the present invention; Figure 3 It is a schematic structural diagram of an electronic device for implementing the knee joint surgical navigation and positioning method based on a neural network combined with medical images provided by an embodiment of the present invention.

[0019] Description of Reference Numerals: 1. Electronic device; 10. Processor; 11. Memory; 12. Bus.

[0020] The implementation, functional features, and advantages of the objectives of the present invention will be further described with reference to the embodiments and the accompanying drawings. Detailed Embodiment

[0021] It should be understood that the specific embodiments described herein are merely used to explain the present invention and are not used to limit the present invention.

[0022] The embodiments of the present application provide a knee joint surgery navigation and positioning method based on a neural network combined with medical images. The execution subject of the knee joint surgery navigation and positioning method based on a neural network combined with medical images includes, but is not limited to, at least one of electronic devices such as a server, a terminal, etc. that can be configured to execute the method provided in the embodiments of the present application. In other words, the knee joint surgery navigation and positioning method based on a neural network combined with medical images can be executed by software or hardware installed on a terminal device or a server device, and the software can be a blockchain platform. The server includes, but is not limited to: a single server, a server cluster, a cloud server, or a cloud server cluster, etc.

[0023] Refer to Figure 1 As shown, it is a flowchart of a knee joint surgery navigation and positioning method based on a neural network combined with medical images provided by an embodiment of the present invention. In this embodiment, the knee joint surgery navigation and positioning method based on a neural network combined with medical images includes: S1. Receive a knee joint surgery instruction, determine the patient to be operated on based on the knee joint surgery instruction, and obtain the information of the patient to be operated on, where the information of the patient to be operated on includes: basic patient information, clinical diagnosis information, and surgical plan information.

[0024] It can be understood that the knee joint surgery instruction refers to an instruction initiated by a person to perform a knee joint surgery on a specific patient, and the patient to be operated on refers to the specific patient indicated in the knee joint surgery instruction. The information of the patient to be operated on refers to the information required to print the knee joint model of the patient to be operated on. Among them, the basic patient information refers to the physiological information of the patient to be operated on, such as: name, age, height, weight, etc. The clinical diagnosis information refers to the clinical information of the patient to be operated on, such as: X-ray films, CT scan information, etc. of the patient to be operated on. The surgical plan information refers to the information of the patient to be operated on in the subsequent knee joint surgery, such as: important parts involved in the surgery: ligaments, menisci, cartilage, etc.

[0025] S2. Construct a surgical printing model based on the information of the patient to be operated on, and perform a three-dimensional scan on the surgical printing model to obtain a three-dimensional knee joint image, where the surgical printing model includes a set of marker balls.

[0026] It should be explained that the surgical printing model refers to the bone model of the knee joint area where the patient to be operated on needs surgery. The construction of the surgical printing model can be achieved through 3D printing technology. The three-dimensional knee joint image refers to the image obtained after performing a CT scan on the surgical printing model. The set of marker balls refers to a set of metal balls used to mark important parts during the operation. The color of the metal balls should be distinguished from other parts (non-metal ball parts) in the surgical printing model for subsequent identification of the marker balls.

[0027] Specifically, the construction of the surgical printing model based on the information of the patient to be operated on includes: Print an initial printing model according to the information of the patient to be operated on; Identify the set of marked parts in the information of the patient to be operated on, obtain the marking positions of each marked part in the set of marked parts, and obtain a set of marking positions; Obtain a set of marker balls based on the set of marked parts, and inlay the set of marker balls on the initial printing model according to the set of marking positions to obtain the surgical printing model.

[0028] It can be understood that the initial printing model refers to the knee joint model of the patient to be operated on obtained after printing, and this model has not been inlaid with marker balls yet. The marked part refers to the knee joint part artificially set for identification in subsequent images, such as the distal femur, proximal tibia, etc. The marking position refers to the specific position of the marked part on the initial printing model. Inlaying the set of marker balls on the initial printing model according to the set of marking positions means inlaying the marker balls on the surface of the marking positions of the initial printing model.

[0029] S3. Perform a two-dimensional scan on the surgical printing model to obtain a set of azimuth knee joint images, where the set of azimuth knee joint images includes multiple azimuth knee joint images, and one marker ball in the set of marker balls is included in the azimuth knee joint image.

[0030] It can be understood that the azimuth knee joint image refers to the two-dimensional photographed image of the surgical printing model, and one marker ball is included in one azimuth knee joint image.

[0031] Specifically, performing a two-dimensional scan on the surgical printing model to obtain a set of azimuth knee joint images includes: Successively extract marker balls from the set of marker balls, set an initial shooting direction according to the marker balls, and determine the surgical shooting method of the patient to be operated on; Based on the surgical shooting method, the initial shooting direction, and the preset single-orientation shooting accuracy, perform two-dimensional shooting on the surgical printing model to obtain a set of original orientation images, where the number of original orientation images in the set of original orientation images is the same as the single-orientation shooting accuracy; Change the shooting orientation of the initial shooting direction to obtain a changed shooting direction, where the shooting orientation change refers to translating or rotating the initial shooting direction; Take the changed shooting direction as the initial shooting direction, and return to the step of performing two-dimensional shooting on the surgical printing model based on the surgical shooting method, the initial shooting direction, and the preset single-orientation shooting accuracy until the number of the set of original orientation images is not less than the preset minimum number of image sets; Merge the set of original orientation images to obtain an orientation knee joint image set.

[0032] It can be understood that the initial shooting direction refers to the direction of shooting the marker ball set manually, for example: shooting the surgical printing model on the same horizontal plane as the marker ball. The surgical shooting method refers to the way the patient to be operated is shot during subsequent knee joint surgery, for example: X-ray shooting. The single-orientation shooting accuracy refers to a constant set manually, which is used to specify the number of times of shooting the surgical printing model in the initial shooting direction. The original orientation image refers to the two-dimensional image obtained by shooting the surgical printing model. The changed shooting direction refers to the initial shooting direction after the orientation change, where the shooting orientation change refers to operations such as rotating and translating the initial shooting direction, so that there is a deviation between the changed shooting direction and the initial shooting direction. The minimum number of image sets refers to a constant set manually, and the merging refers to putting the original orientation images in all sets of original orientation images into a set. The set after putting is the orientation knee joint image set. Here, the sets of original orientation images merged are all sets of original orientation images of all marker balls.

[0033] It should be explained that since a large amount of data (the above-mentioned original orientation images) needs to be input for subsequent deep learning, in order to ensure the integrity and diversity of the data, it is necessary to shoot the surgical printing model in multiple different shooting directions. In addition to the above shooting orientation change, the integrity and diversity of the data can also be increased by image augmentation of the set of original orientation images, where the image augmentation refers to changing the lighting conditions in the original orientation image, performing geometric transformation, color transformation, noise addition, etc. on the original orientation image.

[0034] S4. Sequentially extract orientation knee joint images from the orientation knee joint image set, and use the preset multi-modal registration system to automatically register the orientation knee joint images and the three-dimensional knee joint images to obtain training registration parameters.

[0035] Understandably, the multi-modal registration system refers to a software tool capable of registering two-dimensional images and three-dimensional images, such as Elastix, NiftyReg, etc. The training registration parameters refer to the rotation matrix and translation vector in the automatic registration process. Among them, the rotation matrix refers to a mathematical tool used to describe the rotation of an object in three-dimensional space. It represents the rotation relationship of one coordinate system relative to another coordinate system. In this solution, this rotation matrix is a 3×3 orthogonal matrix. The translation vector refers to a vector used to describe the translation of an object in space. It represents the offset of a point moving from one position to another position.

[0036] S5. Summarize the training registration parameters to obtain a training registration parameter set, and train multiple pre-acquired neural network models according to the training registration parameter set to obtain multiple adaptive registration models.

[0037] Understandably, the neural network model refers to a deep learning model capable of automatically identifying features of the input image. A convolutional neural network can be used as this neural network model, such as ResNet model, EfficientNet model, etc. The adaptive registration model refers to a neural network model after training. Input the two-dimensional image and the three-dimensional image into this adaptive registration model, and this adaptive registration model will output the registration parameters between these two images, that is, output a rotation matrix and a translation vector.

[0038] Furthermore, the training refers to the operation of training the neural network model using the training registration parameters and the corresponding azimuth knee joint image and three-dimensional knee joint image.

[0039] It should be explained that before the operation, first, three-dimensional knee joint images of the patient to be operated on are obtained. These images contain multiple marker balls. The function of these marker balls is to accurately register the two-dimensional images (i.e., the azimuth knee joint images) onto the three-dimensional knee joint images in the subsequent steps. Subsequently, two-dimensional knee joint images of the patient to be operated on are taken. Each of these two-dimensional images contains a marker ball. Through the marker balls, we register the two-dimensional images with the three-dimensional images and record the registration parameters in this process. By repeating the above steps, a large number of registration parameters can be collected. Using these rich registration parameter data, the neural network is trained to construct a registration model that can quickly obtain the registration parameters between the three-dimensional image and the two-dimensional image, that is, the subsequent integrated registration model. This model can efficiently complete the registration task of two images in different dimensions. After completing the above preoperative preparation work, the registration of the two-dimensional image can be quickly completed during the operation. During the operation, two-dimensional knee joint images (i.e., the subsequent intraoperative two-dimensional images) of the patient to be operated on (at this time, the patient to be operated on is undergoing the operation) are taken. In order for the surgeon to accurately understand the specific position of the current surgical site in the entire knee joint, it is necessary to register the two-dimensional knee joint image into the three-dimensional knee joint image obtained before the operation. Different from the traditional direct use of a multimodal registration system for registration, here the preoperatively constructed registration model is used for registration. Since traditional multimodal registration systems such as Elastix and NiftyReg achieve image registration through optimization algorithms, this makes the multimodal registration system have a large delay in real-time intraoperative applications. By constructing a registration model before the operation, the surgeon can quickly and accurately determine the position of the current surgical site based on the two-dimensional image during the operation, greatly improving the efficiency of image registration, thereby improving the accuracy and safety of the operation.

[0040] S6. Model error analysis is performed on multiple adaptive registration models to obtain multiple registration error values and multiple optimal registration marker balls.

[0041] It can be understood that the registration error value refers to the difference value between the registration parameters generated by the adaptive registration model and the actual registration parameters. The optimal registration marker ball refers to the marker ball in the two-dimensional image with the smallest error value in registering with the three-dimensional knee joint image. Among them, the two-dimensional image refers to the azimuth knee joint image. Since the marker balls are used to identify different parts of the knee joint, the knee joint part corresponding to the optimal registration marker ball is the part that the adaptive registration model can predict most accurately.

[0042] Specifically, the model error analysis is performed on multiple adaptive registration models to obtain multiple registration error values and multiple optimal registration marker balls, including: Select a test orientation image group from the azimuth knee joint image set, and pair each test orientation image in the test orientation image group with the three-dimensional knee joint image to obtain a plurality of test knee joint image groups. Among them, each test knee joint image group includes a test orientation image and a three-dimensional knee joint image; Extract the test knee joint image groups in sequence from the plurality of test knee joint image groups, and perform the following operations on the test knee joint image groups: Extract the adaptive registration models in sequence from the plurality of adaptive registration models, input the test knee joint image group into the adaptive registration model to obtain test registration parameters, where the test registration parameters include: rotation matrix and translation vector; Identify the three-dimensional center coordinates and two-dimensional center coordinates in the test knee joint image group, and calculate the unit error value based on the test registration parameters, three-dimensional center coordinates and two-dimensional center coordinates; Summarize the unit error values to obtain a unit error value group, and based on the unit error value group, determine the registration error value and the optimal registration marker ball. Summarize the registration error value and the optimal registration marker ball respectively to obtain a plurality of registration error values and a plurality of optimal registration marker balls.

[0043] It can be understood that the test orientation image group refers to the image combination used to test the adaptive registration model. The test orientation image group can be randomly selected from the azimuth knee joint image set. This random selection can be manually selected or selected by a program. The test knee joint image group refers to the combination of the three-dimensional knee joint image and the test orientation image. Among them, pairing refers to the operation of putting the test orientation image and the three-dimensional knee joint image into an array. The test registration parameters refer to the output of the adaptive registration model. The three-dimensional center coordinates refer to the coordinates of the marker ball in the three-dimensional knee joint image in the test knee joint image group, and the two-dimensional center coordinates refer to the coordinates of the marker ball in the two-dimensional knee joint image in the test knee joint image group. It should be noted that the three-dimensional center coordinates and the two-dimensional center coordinates should be the coordinates of the same marker ball in different images. The unit error value refers to the error value of the adaptive registration model when registering the test knee joint image group.

[0044] Specifically, the identifying the three-dimensional center coordinates and the two-dimensional center coordinates in the test knee joint image group includes: Identify the test three-dimensional image and the test two-dimensional image in the test knee joint image group; Obtain the binary image of the test three-dimensional image, perform a connected component operation on the binary image to obtain a connected component group; Determine the pixel volume of the marked ball, set the marked ball volume range according to the pixel volume, and filter the connected domain group by using the marked ball volume range to obtain an effective connected domain group, where the effective connected domain group includes multiple effective connected domains, and the volume of the effective connected domain is within the marked ball volume range; Extract the effective connected domains from the effective connected domain group in sequence, determine the boundary image of the effective connected domain, and perform a three-dimensional Hough transform on the boundary image to obtain a transformed image; Identify the maximum pixel point in the transformed image, where the maximum pixel point is the pixel point with the largest pixel value in the transformed image; Summarize the maximum pixel points to obtain a maximum pixel point group, and determine a marked ball pixel point group in the maximum pixel point group based on a preset number of marked balls, where the number of marked ball pixel points in the marked ball pixel point group is the same as the number of marked balls; Obtain the marked ball coordinate group of the marked ball pixel point group; Determine the two-dimensional marked ball and the two-dimensional ball center coordinates in the test two-dimensional image, and determine the three-dimensional ball center coordinates corresponding to the two-dimensional marked ball in the marked ball coordinate group.

[0045] It can be understood that the test three-dimensional image and the test two-dimensional image respectively refer to the azimuth knee joint image and the three-dimensional knee joint image in the test knee joint image group. The connected domain operation on the binary image refers to the process of identifying and marking the pixel regions connected to each other in the image, splitting the image into multiple independent connected domains, and the connected domain operation has wide applications in the fields of image processing, computer graphics, and computer vision, etc., which will not be elaborated here. The pixel volume refers to the volume of the marked ball in the three-dimensional knee joint image, and the marked ball volume range refers to the range set artificially, which is used to filter the connected domains in the connected domain group. Among them, filtering means that when the volume of the connected domain is not within the marked ball volume range, it means that the connected domain is not the pixel region corresponding to the marked ball, that is, the connected domain is removed from the connected domain group. The effective connected domain group refers to the connected domain group after filtering. The boundary image refers to the image representing the boundary in the effective connected domain, and determining the boundary image of the effective connected domain means using an edge detection algorithm to detect the boundary image in the effective connected domain. Among them, the edge detection algorithm is, for example, Canny edge detection, etc. The transformed image refers to the boundary image after the three-dimensional Hough transform. Among them, the three-dimensional Hough transform refers to detecting a specific geometric shape in the boundary image. In this step, the specific geometric shape is a sphere, and this step of the three-dimensional Hough transform can be implemented by a three-dimensional Hough transform tool.

[0046] It should be explained that the number index of the marked balls refers to the number of marked balls in the marked ball group. The marked ball pixel point group refers to the combination of the r pixel points with the largest pixel values in the transformed image, where r represents the number of marked balls. The marked ball coordinates refer to the coordinates of the marked ball pixel points in the three-dimensional knee joint image, and one marked ball coordinate corresponds to only one marked ball.

[0047] Further, the two-dimensional marked balls and the two-dimensional ball center coordinates respectively refer to the marked balls and the coordinates of the corresponding marked balls in the test two-dimensional image. The determination method of the two-dimensional ball center coordinates is similar to the acquisition method of the above three-dimensional spherical coordinates, only changing the three-dimensional Hough transform to a two-dimensional Hough transform.

[0048] Specifically, calculating the unit error value based on the test registration parameters, the three-dimensional ball center coordinates, and the two-dimensional ball center coordinates includes: Identifying the two-dimensional shooting device and constructing the internal parameter matrix of the two-dimensional shooting device, where the internal parameter matrix is expressed as: Where, represents the internal parameter matrix, and respectively represent the focal lengths of the two-dimensional shooting device on the x-axis and y-axis, and respectively represent the abscissa and ordinate in the two-dimensional ball center coordinates; Calculating the transformed three-dimensional coordinates according to the test registration parameters, the internal parameter matrix, and the three-dimensional ball center coordinates, where the transformed three-dimensional coordinates are expressed as: Where, represents the transformed three-dimensional coordinates, represents the abscissa in the transformed three-dimensional coordinates, represents the ordinate in the transformed three-dimensional coordinates, represents the vertical coordinate in the transformed three-dimensional coordinates, represents the rotation matrix in the test registration parameters, represents the three-dimensional ball center coordinates, represents the abscissa in the three-dimensional ball center coordinates, represents the ordinate in the three-dimensional ball center coordinates, represents the vertical coordinate in the three-dimensional ball center coordinates, represents the translation vector in the test registration parameters, represents the symbol of the vector product; Calculating the unit error value according to the transformed three-dimensional coordinates and the two-dimensional ball center coordinates.

[0049] It is understandable that the two-dimensional imaging device refers to the imaging device required for the surgical imaging method. The converted three-dimensional coordinates refer to the coordinates on a two-dimensional plane. Since the converted three-dimensional coordinates are three-dimensional coordinates, it is necessary to project the three-dimensional coordinates onto a two-dimensional plane to obtain two-dimensional converted coordinates.

[0050] Specifically, calculating the unit error value according to the converted three-dimensional coordinates and the two-dimensional spherical center coordinates includes: Performing a two-dimensional projection on the converted three-dimensional coordinates to obtain two-dimensional converted coordinates, where the two-dimensional converted coordinates are expressed as: Wherein, represents the two-dimensional converted coordinates, represents the abscissa in the two-dimensional converted coordinates, represents the ordinate in the two-dimensional converted coordinates; According to the two-dimensional converted coordinates and the two-dimensional spherical center coordinates, calculate the unit error value using the following formula: Wherein, represents the unit error value.

[0051] It is understandable that the two-dimensional converted coordinates refer to the coordinates obtained after two-dimensional projection of the converted three-dimensional coordinates, and the two-dimensional projection refers to converting the three-dimensional converted coordinates into two-dimensional two-dimensional converted coordinates.

[0052] Specifically, determining the registration error value and the optimal registration marker ball based on the unit error value group includes: Identifying the error marker balls corresponding to the unit error values in the unit error value group to obtain an error marker ball group, where the error marker balls are the marker balls corresponding to the two-dimensional spherical center coordinates in the formula for calculating the unit error value; Classifying the unit error value group according to the error marker ball group to obtain multiple groups of similar error values, where in the same group of similar error values, all similar error values correspond to the same error marker ball; Setting the marker ball weight group of the error marker ball group, calculating the average similar error value of each group of similar error values in the multiple groups of similar error values to obtain an average similar error value group; Calculating the registration error value according to the marker ball weight group and the average similar error value group, where the registration error value is expressed as: Wherein, represents the registration error value, represents the number of groups of similar error values in the multiple groups of similar error values, represents the th marker ball weight in the marker ball weight group, represents the th average similar error value in the average similar error value group; Identify the optimal error value in the average similar error value group, where the optimal error value is the average similar error value with the smallest numerical value in the average similar error value group; Identify the optimal registration marker ball corresponding to the optimal error value in the error marker ball group.

[0053] Understandably, the error marker ball refers to the marker ball corresponding to the two-dimensional spherical center coordinates in the formula for calculating the unit error value. Classifying the unit error value group means placing the unit error values corresponding to the same error marker ball in the same array, and this array is the similar error value group. The marker ball weight group refers to the constant of each error marker ball set artificially. Since different error marker balls represent different knee joint parts, and different knee joint parts have different importance in specific surgeries, the marker ball weight group can be set to represent the importance of different knee joint parts in the surgery. The average similar error value refers to the average of all similar error values in the similar error value group.

[0054] S7. Integrate the multiple adaptive registration models according to the multiple registration error values and the multiple optimal registration marker balls to obtain an integrated registration model.

[0055] It should be explained that in order to ensure that the registration parameters obtained in actual surgeries have sufficient accuracy, multiple neural network models are selected for training. Since a single neural network model has deficiencies in terms of robustness, by training multiple adaptive registration models, the advantages of each adaptive registration model can be fully utilized. The registration errors of these adaptive registration models vary in different knee joint parts, that is, the optimal registration marker balls corresponding to different adaptive registration models are different. Therefore, model integration is performed according to the error performance of each adaptive registration model in different parts. In this way, an integrated registration model can be obtained, and this integrated registration model can provide higher accuracy in intraoperative image registration, thus greatly improving the reliability of surgical navigation.

[0056] Specifically, integrating the multiple adaptive registration models to obtain an integrated registration model includes: Extract the adaptive registration model from the multiple adaptive registration models in sequence, and identify the model error value of the adaptive registration model among the multiple registration error values; Confirm the optimal registration marker ball of the adaptive registration model, and identify the optimal registration weight corresponding to the optimal registration marker ball in the marker ball weight group; According to the model error value and the optimal registration weight, calculate the model integration weight using the following formula: Where, represents the model integration weight, represents the optimal registration weight, represents the natural constant, Represents the model error value, represents the th registration error value among multiple registration error values, represents the number of registration error values among multiple registration error values; Summarize the model integration weights to obtain multiple model integration weights. According to the multiple model integration weights, integrate multiple adaptive registration models to obtain an integrated registration model.

[0057] It is understandable that the model error value refers to the registration error value of the adaptive registration model, the optimal registration weight refers to the marker ball weight corresponding to the optimal registration marker ball, and the model integration weight refers to the weight of the adaptive registration model during subsequent integration. Further, since different adaptive registration models have different registration capabilities for different knee joint parts, that is, the unit error values corresponding to different marker balls are different, when calculating the model integration weight, the marker ball weight corresponding to the knee joint part with the strongest registration ability of the adaptive registration model should be considered.

[0058] It should be explained that the integrated registration model is an overall model that integrates the outputs of multiple adaptive registration models into one output value through multiple model integration weights. Among them, the output of the integrated registration model is: Among them, represents the rotation matrix output by the integrated registration model, represents the th model integration weight among multiple model integration weights, represents the rotation matrix output by the th adaptive registration model among multiple adaptive registration models, represents the translation vector output by the integrated registration model, represents the th model integration weight, represents the th translation vector output by the

[0059] S8. Obtain intraoperative two-dimensional images, input the intraoperative two-dimensional images and three-dimensional knee joint images into the integrated registration model to obtain three-dimensional navigation positioning images, and complete the knee joint surgery navigation positioning based on the neural network combined with medical images based on the three-dimensional navigation positioning images.

[0060] It is understandable that the intraoperative two-dimensional image refers to the two-dimensional knee joint image of the patient to be operated on taken in real time during knee joint surgery. The intraoperative two-dimensional image and the three-dimensional knee joint image are input into the integrated registration model, and the model will output a rotation matrix and a translation vector. According to the rotation matrix and the translation vector, the intraoperative two-dimensional image and the three-dimensional knee joint image can be registered. After registration, a three-dimensional navigation positioning image can be obtained, and the three-dimensional navigation positioning image refers to the three-dimensional knee joint image containing the intraoperative two-dimensional image.

[0061] To solve the problems described in the background art, the present invention constructs a surgical printing model and performs three-dimensional scanning, which can generate a high-precision three-dimensional knee joint image, providing an intuitive three-dimensional anatomical structure for surgical navigation. The marker ball group on the surgical printing model provides accurate reference points for subsequent image registration, ensuring that the two-dimensional image can be accurately aligned to the three-dimensional image. Then, by two-dimensionally scanning the surgical printing model, azimuth knee joint images in multiple orientations can be obtained. These azimuth knee joint images show the anatomical structure of the knee joint from different angles, thus providing diverse data for the training of the subsequent neural network model, improving the generalization ability and robustness of the model. Further, using a multi-modal registration system, the azimuth knee joint images and the three-dimensional knee joint images are automatically registered to obtain training registration parameters. These training registration parameters provide key data support for the training of the neural network model. Multiple neural network models are trained according to the training registration parameter set to obtain multiple adaptive registration models. These adaptive registration models can automatically adapt to different image features and transformation relationships by learning a large number of registration parameters, thus quickly and accurately completing image registration during the operation. The training of multiple models not only improves the registration accuracy but also enhances the robustness of the system, enabling it to handle complex clinical scenarios and individual differences. Then, through model error analysis, the performance of each adaptive registration model is evaluated to obtain the registration error value and the optimal registration marker ball of each adaptive registration model. These registration error values and optimal registration marker balls provide important reference bases for subsequent model integration. By integrating multiple adaptive registration models, the advantages of each model can be combined to generate an integrated registration model with better performance. The integrated registration model can make full use of the registration advantages of each adaptive registration model in different knee joint parts, reduce the limitations of a single model, and improve the overall registration accuracy and robustness, thereby significantly improving the accuracy and reliability of surgical navigation. Finally, the intraoperative two-dimensional image and the three-dimensional knee joint image are input into the integrated registration model, and a three-dimensional navigation positioning image can be quickly generated, providing real-time navigation information for the surgeon and greatly reducing the time for intraoperative image registration. Therefore, the present invention can improve the accuracy of navigation positioning during knee joint surgery and reduce the time for image registration during knee joint surgery.

[0062] Such as Figure 2As shown in the figure, it is a functional block diagram of a knee joint surgery navigation and positioning system based on a neural network combined with medical images provided by an embodiment of the present invention.

[0063] The knee joint surgery navigation and positioning system 100 based on a neural network combined with medical images according to the present invention can be installed in an electronic device. According to the functions achieved, the knee joint surgery navigation and positioning system 100 based on a neural network combined with medical images can include a three-dimensional image scanning module 101, a training parameter acquisition module 102, a model error analysis module 103, and an intraoperative image registration module 104. The modules in the present invention can also be referred to as units, which refer to a series of computer program segments that can be executed by a processor of an electronic device and can complete fixed functions, and are stored in the memory of the electronic device.

[0064] The three-dimensional image scanning module 101 is configured to receive a knee joint surgery instruction, determine a patient to be operated on based on the knee joint surgery instruction, and obtain information of the patient to be operated on, wherein the information of the patient to be operated on includes: basic patient information, clinical diagnosis information, and surgical plan information. A surgical printing model is constructed according to the information of the patient to be operated on, and the surgical printing model is three-dimensionally scanned to obtain a three-dimensional knee joint image, wherein the surgical printing model includes a set of marker balls; The training parameter acquisition module 102 is configured to two-dimensionally scan the surgical printing model to obtain a set of azimuth knee joint images, wherein the set of azimuth knee joint images includes multiple azimuth knee joint images, and one marker ball in the set of marker balls is included in the azimuth knee joint image. The azimuth knee joint images are sequentially extracted from the set of azimuth knee joint images, and the azimuth knee joint images and the three-dimensional knee joint image are automatically registered by using a preset multimodal registration system to obtain training registration parameters; The model error analysis module 103 is configured to summarize the training registration parameters to obtain a set of training registration parameters, train a plurality of pre-acquired neural network models according to the set of training registration parameters to obtain a plurality of adaptive registration models, and perform model error analysis on the plurality of adaptive registration models to obtain a plurality of registration error values and a plurality of optimal registration marker balls; The intraoperative image registration module 104 is configured to integrate the plurality of adaptive registration models according to the plurality of registration error values and the plurality of optimal registration marker balls to obtain an integrated registration model, obtain an intraoperative two-dimensional image, and input the intraoperative two-dimensional image and the three-dimensional knee joint image into the integrated registration model to obtain a three-dimensional navigation and positioning image.

[0065] Specifically, each module in the knee joint surgery navigation and positioning system 100 based on a neural network combined with medical images in the embodiment of the present invention adopts the same as the above-mentioned Figure 1It uses the same technical means as the knee joint surgery navigation and positioning method based on neural network combined with medical images described in [reference], and can produce the same technical effects, which will not be elaborated here.

[0066] As Figure 3 shown, it is a schematic structural diagram of an electronic device for implementing the knee joint surgery navigation and positioning method based on neural network combined with medical images according to an embodiment of the present invention.

[0067] The electronic device 1 may include a processor 10, a memory 11, and a bus 12, and may further include a computer program stored in the memory 11 and executable on the processor 10, such as a program for the knee joint surgery navigation and positioning method based on neural network combined with medical images.

[0068] Among them, the memory 11 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, mobile hard disk, multimedia card, card-type memory (such as SD or DX memory, etc.), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 11 may be an internal storage unit of the electronic device 1, such as the mobile hard disk of the electronic device 1. In other embodiments, the memory 11 may also be an external storage device of the electronic device 1, such as a plug-in mobile hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. equipped on the electronic device 1. Further, the memory 11 also includes the internal storage unit of the electronic device 1 and the external storage device. The memory 11 can not only be used to store application software installed in the electronic device 1 and various types of data, such as the code of the program for the knee joint surgery navigation and positioning method based on neural network combined with medical images, but also be used to temporarily store data that has been output or will be output.

[0069] In some embodiments, the processor 10 may be composed of integrated circuits. For example, it may be composed of a single packaged integrated circuit, or may be composed of multiple integrated circuits with the same or different functions, including a combination of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 10 is the control core (Control Unit) of the electronic device, connecting various components of the entire electronic device through various interfaces and lines, and by running or executing programs or modules stored in the memory 11 (such as the program for the knee joint surgery navigation and positioning method based on neural network combined with medical images, etc.), and calling data stored in the memory 11, to execute various functions of the electronic device 1 and process data.

[0070] The bus 12 may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The bus 12 may be divided into an address bus, a data bus, a control bus, etc. The bus 12 is configured to implement connection communication between the memory 11 and at least one processor 10, etc.

[0071] Figure 3 Only the electronic device with components is shown. Those skilled in the art can understand that Figure 3 the shown structure does not constitute a limitation on the electronic device 1, and it may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0072] For example, although not shown, the electronic device 1 may further include a power source (such as a battery) for powering each component. Preferably, the power source may be logically connected to the at least one processor 10 through a power management system, so as to implement functions such as charge management, discharge management, and power consumption management through the power management system. The power source may also include any components such as one or more DC or AC power sources, a recharge system, a power failure detection circuit, a power converter or inverter, a power status indicator, etc. The electronic device 1 may also include a variety of sensors, a Bluetooth module, a Wi-Fi module, etc., which will not be elaborated here.

[0073] Furthermore, the electronic device 1 may further include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is generally used to establish a communication connection between the electronic device 1 and other electronic devices.

[0074] Optionally, the electronic device 1 may further include a user interface. The user interface may be a display, an input unit (such as a keyboard), and optionally, the user interface may also be a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch liquid crystal display, and an OLED (Organic Light-Emitting Diode) toucher, etc. Among them, the display may also be appropriately referred to as a display screen or a display unit, which is used to display the information processed in the electronic device 1 and to display a visual user interface.

[0075] The program of the knee joint surgery navigation and positioning method based on neural network combined with medical images stored in the memory 11 in the electronic device 1 is a combination of multiple instructions. When running in the processor 10, it can achieve: Receive a knee joint surgery instruction, determine the patient to be operated on based on the knee joint surgery instruction, and obtain the information of the patient to be operated on, where the information of the patient to be operated on includes: basic patient information, clinical diagnosis information, and surgical plan information; Construct a surgical printing model according to the information of the patient to be operated on, perform three-dimensional scanning on the surgical printing model to obtain a three-dimensional knee joint image, where the surgical printing model includes a set of marker balls; Perform two-dimensional scanning on the surgical printing model to obtain a set of azimuth knee joint images, where the set of azimuth knee joint images includes multiple azimuth knee joint images, and one marker ball in the set of marker balls is included in the azimuth knee joint image; Sequentially extract the azimuth knee joint images from the set of azimuth knee joint images, and use a preset multi-modal registration system to automatically register the azimuth knee joint images and the three-dimensional knee joint image to obtain training registration parameters; Summarize the training registration parameters to obtain a set of training registration parameters, and train a plurality of pre-obtained neural network models according to the set of training registration parameters to obtain a plurality of adaptive registration models; Perform model error analysis on a plurality of adaptive registration models to obtain a plurality of registration error values and a plurality of optimal registration marker balls; Integrate the plurality of adaptive registration models according to the plurality of registration error values and the plurality of optimal registration marker balls to obtain an integrated registration model; Obtain intraoperative two-dimensional images, input the intraoperative two-dimensional images and the three-dimensional knee joint image into the integrated registration model to obtain a three-dimensional navigation and positioning image, and complete the knee joint surgery navigation and positioning based on neural network combined with medical images based on the three-dimensional navigation and positioning image.

[0076] Specifically, the specific implementation method of the above instructions by the processor 10 can refer to Figures 1 to 3 The description of the relevant steps in the corresponding embodiment, which will not be repeated here.

[0077] Furthermore, if the modules / units integrated in the electronic device 1 are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile. For example, the computer-readable medium can include: any entity or system capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disc, computer memory, read-only memory (ROM, Read-Only Memory).

[0078] The present invention also provides a computer-readable storage medium storing a computer program, which when executed by a processor of an electronic device, can implement: Receiving a knee joint surgery instruction, determining a patient to be operated on based on the knee joint surgery instruction, and obtaining the information of the patient to be operated on, wherein the information of the patient to be operated on includes: basic patient information, clinical diagnosis information, and surgical plan information; Constructing a surgical printing model according to the information of the patient to be operated on, performing three-dimensional scanning on the surgical printing model to obtain a three-dimensional knee joint image, wherein the surgical printing model includes a set of marker balls; Performing two-dimensional scanning on the surgical printing model to obtain a set of azimuth knee joint images, wherein the set of azimuth knee joint images includes multiple azimuth knee joint images, and one marker ball in the set of marker balls is included in the azimuth knee joint image; Sequentially extracting the azimuth knee joint images from the set of azimuth knee joint images, and automatically registering the azimuth knee joint images and the three-dimensional knee joint image by using a preset multi-modal registration system to obtain training registration parameters; Summarizing the training registration parameters to obtain a set of training registration parameters, and training a plurality of pre-obtained neural network models according to the set of training registration parameters to obtain a plurality of adaptive registration models; Performing model error analysis on each of the plurality of adaptive registration models to obtain a plurality of registration error values and a plurality of optimal registration marker balls; Integrating the plurality of adaptive registration models according to the plurality of registration error values and the plurality of optimal registration marker balls to obtain an integrated registration model; Obtaining an intraoperative two-dimensional image, inputting the intraoperative two-dimensional image and the three-dimensional knee joint image into the integrated registration model to obtain a three-dimensional navigation positioning image, and completing the knee joint surgery navigation positioning based on the neural network combined with medical images based on the three-dimensional navigation positioning image.

[0079] In several embodiments provided by the present invention, it should be understood that the disclosed devices, systems, and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative, and there may be other partitioning methods in actual implementation.

[0080] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0081] In addition, in each of the embodiments of the present invention, the functional modules may be integrated into a processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware, or in the form of a combination of hardware and software functional modules.

[0082] For those skilled in the art, it is obvious that the present invention is not limited to the details of the above exemplary embodiments, and without departing from the spirit or basic characteristics of the present invention, the present invention can be implemented in other specific forms.

[0083] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be modified or equivalently replaced without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A knee joint surgery navigation and positioning method based on a neural network combined with medical images, characterized in that, The method includes: Receiving a knee joint surgery instruction, determining a patient to be operated on based on the knee joint surgery instruction, and obtaining the information of the patient to be operated on, where the information of the patient to be operated on includes: basic patient information, clinical diagnosis information, and surgical plan information; Constructing a surgical printing model according to the information of the patient to be operated on, performing three-dimensional scanning on the surgical printing model to obtain a three-dimensional knee joint image, where the surgical printing model includes a set of marker balls; Performing two-dimensional scanning on the surgical printing model to obtain a set of azimuth knee joint images, where the set of azimuth knee joint images includes multiple azimuth knee joint images, and one marker ball in the set of marker balls is included in the azimuth knee joint image; Sequentially extracting the azimuth knee joint images from the set of azimuth knee joint images, and using a preset multi-modal registration system to automatically register the azimuth knee joint images and the three-dimensional knee joint image to obtain training registration parameters; Summarizing the training registration parameters to obtain a set of training registration parameters, and training a plurality of pre-obtained neural network models according to the set of training registration parameters to obtain a plurality of adaptive registration models; Performing model error analysis on each of the plurality of adaptive registration models to obtain a plurality of registration error values and a plurality of optimal registration marker balls; Integrating the plurality of adaptive registration models according to the plurality of registration error values and the plurality of optimal registration marker balls to obtain an integrated registration model; Obtaining an intraoperative two-dimensional image, inputting the intraoperative two-dimensional image and the three-dimensional knee joint image into the integrated registration model to obtain a three-dimensional navigation positioning image, and completing the knee joint surgery navigation positioning based on the neural network combined with medical images based on the three-dimensional navigation positioning image.

2. The knee joint surgery navigation and positioning method based on neural network combined with medical images according to claim 1, characterized in that, The constructing a surgical printing model according to the information of the patient to be operated on includes: Printing an initial printing model according to the information of the patient to be operated on; Identifying a set of marked parts in the information of the patient to be operated on, obtaining the marked positions of each marked part in the set of marked parts to obtain a set of marked positions; Obtaining a set of marker balls based on the set of marked parts, and inlaying the set of marker balls on the initial printing model according to the set of marked positions to obtain a surgical printing model.

3. The knee joint surgery navigation and positioning method based on neural network combined with medical images according to claim 2, wherein The performing two-dimensional scanning on the surgical printing model to obtain a set of azimuth knee joint images includes: Sequentially extracting marker balls from the set of marker balls, setting an initial shooting direction according to the marker balls, and determining the surgical shooting method of the patient to be operated on; Performing two-dimensional shooting on the surgical printing model based on the surgical shooting method, the initial shooting direction, and a preset single-azimuth shooting accuracy to obtain a set of original azimuth images, where the number of original azimuth images in the set of original azimuth images is the same as the single-azimuth shooting accuracy; Changing the shooting azimuth of the initial shooting direction to obtain a changed shooting direction, where the shooting azimuth change refers to translating or rotating the initial shooting direction; Taking the changed shooting direction as the initial shooting direction, and returning to the step of performing two-dimensional shooting on the surgical printing model based on the surgical shooting method, the initial shooting direction, and the preset single-azimuth shooting accuracy until the number of the set of original azimuth images is not less than a preset minimum number of image groups; Merging the set of original azimuth images to obtain a set of azimuth knee joint images.

4. The knee joint surgery navigation and positioning method based on neural network combined with medical images according to claim 3, wherein, Performing model error analysis on multiple adaptive registration models to obtain multiple registration error values and multiple optimal registration fiducial spheres, including: Selecting a test azimuth image group from the azimuth knee image set, and pairing each test azimuth image in the test azimuth image group with the three-dimensional knee image to obtain multiple test knee image groups, where a test knee image group includes a test azimuth image and a three-dimensional knee image; Successively extracting the test knee image groups from the multiple test knee image groups, and performing the following operations on the test knee image groups: Successively extracting the adaptive registration models from the multiple adaptive registration models, and inputting the test knee image group into the adaptive registration model to obtain test registration parameters, where the test registration parameters include: a rotation matrix and a translation vector; Identifying the three-dimensional center coordinates and two-dimensional center coordinates in the test knee image group, and calculating the unit error value based on the test registration parameters, the three-dimensional center coordinates, and the two-dimensional center coordinates; Summarizing the unit error values to obtain a unit error value group, determining the registration error value and the optimal registration fiducial sphere based on the unit error value group, respectively summarizing the registration error value and the optimal registration fiducial sphere to obtain multiple registration error values and multiple optimal registration fiducial spheres.

5. The knee joint surgery navigation and positioning method based on neural network combined with medical images according to claim 4, wherein, The identifying the three-dimensional center coordinates and two-dimensional center coordinates in the test knee image group includes: Identifying the test three-dimensional image and the test two-dimensional image in the test knee image group; Obtaining the binary image of the test three-dimensional image, performing a connected component operation on the binary image to obtain a connected component group; Determining the pixel volume of the fiducial sphere, setting a fiducial sphere volume range according to the pixel volume, and filtering the connected component group by using the fiducial sphere volume range to obtain an effective connected component group, where the effective connected component group includes multiple effective connected components, and the volume of the effective connected component is within the fiducial sphere volume range; Successively extracting the effective connected components from the effective connected component group, determining the boundary image of the effective connected component, and performing a three-dimensional Hough transform on the boundary image to obtain a transformed image; Identifying the maximum pixel point in the transformed image, where the maximum pixel point is the pixel point with the largest pixel value in the transformed image; Summarizing the maximum pixel points to obtain a maximum pixel point group, and determining a fiducial sphere pixel point group in the maximum pixel point group based on the preset number of fiducial spheres, where the number of fiducial sphere pixel points in the fiducial sphere pixel point group is the same as the number of fiducial spheres; Obtaining the fiducial sphere coordinate group of the fiducial sphere pixel point group; Determining the two-dimensional fiducial sphere and the two-dimensional center coordinates in the test two-dimensional image, and determining the three-dimensional center coordinates corresponding to the two-dimensional fiducial sphere in the fiducial sphere coordinate group.

6. The knee joint surgery navigation and positioning method based on neural network combined with medical images according to claim 5, characterized in that, The calculating the unit error value based on the test registration parameters, the three-dimensional center coordinates, and the two-dimensional center coordinates includes: Identify the two-dimensional imaging device and construct the internal parameter matrix of the two-dimensional imaging device, where the internal parameter matrix is expressed as: Where represents the internal parameter matrix, and respectively represent the focal lengths of the two-dimensional imaging device on the x-axis and y-axis, and respectively represent the abscissa and ordinate in the two-dimensional spherical center coordinates; Calculate the transformed three-dimensional coordinates according to the test registration parameters, the internal parameter matrix, and the three-dimensional center-of-sphere coordinates, where the transformed three-dimensional coordinates are expressed as: where represents the transformed three-dimensional coordinates, represents the abscissa in the transformed three-dimensional coordinates, represents the ordinate in the transformed three-dimensional coordinates, represents the vertical coordinate in the transformed three-dimensional coordinates, represents the rotation matrix in the test registration parameters, represents the three-dimensional center-of-sphere coordinates, represents the abscissa in the three-dimensional center-of-sphere coordinates, represents the ordinate in the three-dimensional center-of-sphere coordinates, represents the vertical coordinate in the three-dimensional center-of-sphere coordinates, represents the translation vector in the test registration parameters, represents the product symbol of vectors; Calculating the unit error value according to the transformed three-dimensional coordinates and the two-dimensional center coordinates.

7. The knee joint surgery navigation and positioning method based on neural network combined with medical images according to claim 6, characterized in that The calculating the unit error value according to the transformed three-dimensional coordinates and the two-dimensional center coordinates includes: Perform a two-dimensional projection on the converted three-dimensional coordinates to obtain two-dimensional converted coordinates, where the two-dimensional converted coordinates are expressed as: Where represents the two-dimensional converted coordinates, represents the abscissa in the two-dimensional converted coordinates, represents the ordinate in the two-dimensional converted coordinates; According to the two-dimensional conversion coordinates and the two-dimensional center-of-sphere coordinates, calculate the unit error value using the following formula: where represents the unit error value.

8. The knee joint surgery navigation and positioning method based on neural network combined with medical images according to claim 7, wherein, The determining the registration error value and the optimal registration fiducial sphere based on the unit error value group includes: Identify the error marker balls corresponding to the unit error values in the unit error value group to obtain an error marker ball group, where the error marker ball is the marker ball corresponding to the two-dimensional spherical center coordinates in the formula for calculating the unit error value; Classify the unit error value group according to the error marker ball group to obtain multiple groups of similar error values. Among them, in the same group of similar error values, all similar error values correspond to the same error marker ball; Set the marker ball weight group of the error marker ball group, calculate the average similar error value of each group of similar error values in the multiple groups of similar error values to obtain an average similar error value group; Calculate the registration error value according to the marked ball right reorganization and the average error value group of the same kind, wherein the registration error value is expressed as: Wherein, represents the registration error value, represents the number of error value groups of the same kind in multiple error value groups of the same kind, represents the th marked ball weight in the marked ball right reorganization, represents the th average error value of the same kind in the average error value group of the same kind; Identify the optimal error value in the average similar error value group, where the optimal error value is the average similar error value with the smallest value in the average similar error value group; Identify the optimal registration marker ball corresponding to the optimal error value in the error marker ball group.

9. The knee joint surgery navigation and positioning method based on neural network combined with medical images according to claim 8, wherein, The integration of the multiple adaptive registration models to obtain an integrated registration model includes: Successively extract the adaptive registration models in the multiple adaptive registration models, and identify the model error values of the adaptive registration models among the multiple registration error values; Confirm the optimal registration marker ball of the adaptive registration model, and identify the optimal registration weight corresponding to the optimal registration marker ball in the marker ball weight group; Calculate the model integration weight according to the model error value and the optimal registration weight using the following formula: where represents the model integration weight, represents the optimal registration weight, represents the natural constant, represents the model error value, represents the th registration error value among multiple registration error values, represents the number of registration error values among multiple registration error values; Summarize the model integration weights to obtain multiple model integration weights, and integrate the multiple adaptive registration models according to the multiple model integration weights to obtain an integrated registration model.

10. A knee joint surgery navigation and positioning system based on a neural network combined with medical images, characterized in that, The system includes: A three-dimensional image scanning module, configured to receive a knee joint surgery instruction, determine a patient to be operated on based on the knee joint surgery instruction, and obtain information of the patient to be operated on, where the information of the patient to be operated on includes: basic patient information, clinical diagnosis information, and surgical plan information. Construct a surgical printing model according to the information of the patient to be operated on, and perform three-dimensional scanning on the surgical printing model to obtain a three-dimensional knee joint image, where the surgical printing model includes a marker ball group; A training parameter acquisition module, configured to perform two-dimensional scanning on the surgical printing model to obtain an azimuth knee joint image set, where the azimuth knee joint image set includes multiple azimuth knee joint images, and one marker ball in the marker ball group is included in the azimuth knee joint image. Extract the azimuth knee joint images in the azimuth knee joint image set in sequence, and use a preset multi-modal registration system to automatically register the azimuth knee joint image and the three-dimensional knee joint image to obtain training registration parameters; A model error analysis module, configured to summarize the training registration parameters to obtain a training registration parameter set, train multiple pre-acquired neural network models according to the training registration parameter set to obtain multiple adaptive registration models, and perform model error analysis on all the multiple adaptive registration models to obtain multiple registration error values and multiple optimal registration marker balls; An intraoperative image registration module, configured to integrate the multiple adaptive registration models according to the multiple registration error values and the multiple optimal registration marker balls to obtain an integrated registration model, obtain an intraoperative two-dimensional image, and input the intraoperative two-dimensional image and the three-dimensional knee joint image into the integrated registration model to obtain a three-dimensional navigation and positioning image.

Images